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Introduction 


Exactly twenty years ago, in July 1999, Nature Neuroscience, then a recently 
launched scientific journal, published a manuscript entitled “Real-time control of a 
robot arm using simultaneously recorded neurons in the motor cortex.” The result of 
one more collaboration between my laboratory and John Chapin’s, this paper launched 
the field of brain-machine interfaces (BMIs) in earnest and caused a major sensation in 
the neuroscience community. Curiously, the name brain-machine interface would only 
appear a year later (Nicolelis, 2001), coined in a review paper entitled “Actions from 
thoughts” that I wrote for Nature, following a request by one of its editors, Charles 
Jennings, who happened to be the first editor-in-chief of Nature Neuroscience. 

To celebrate the twenty-year anniversary of that landmark paper, I have decided 
to collate all BMI-related articles published by my lab at the Duke University Center for 
Neuroengineering, as well as the Edmond and Lily Safra International Institute of 
Neuroscience and the neurorehabilitation laboratory that I founded in Brazil during the 
past two decades. The aim of this initiative is to place in two volumes, the first of a 
series entitled the Nicolelis Lab Series, the entire library of manuscripts that my students 
and I have produced as a result of our research in the BMI field. To some degree, these 
volumes are also historic documents that show the evolution of our ideas in the area as 
well as the many “spin offs” that emerged from our tinkering with the original — and 
now classic — concept of BMIs. That included our transition from using this approach 
primarily as a basic science tool to probe the brain, in search of the physiological 
principles governing the behavior of neuronal circuits — or neural ensembles, as I like 
to call them — to our current major drive to develop assistive and even therapeutic 
strategies for neurological and psychiatric disorders that incorporate BMIs, either alone 
or in conjunction with other technologies (e.g. virtual reality and robotics) and clinical 
approaches. 

To help seasoned practitioners and newcomers alike, I have divided the 
manuscripts that form these two volumes in multiple categories, which loosely cover 
some of the history of the BMI field, at least the way I see it, while offering a reasonable 
way to reconstruct how the rational and experimental work performed in my labs 
evolved until it reached its current focus on developing multiple BMI-based therapies. 

Following this basic structure, Volume 1 begins with a collection of manuscripts 
that I deem the “Foundation” papers. This section contains some of the key studies 
executed, initially when I was a postdoc in John Chapin’s lab, and then by my own lab 
at Duke University that led to the maturation and optimization of the fundamental 
experimental methods employed in the invention and dissemination of BMI during its 
first fifteen years (Nicolelis et al., 1995). I am referring to the neurophysiological 
approach known as chronic, multi-site, multi-electrode recordings in behaving animals, 
which John Chapin and I originally developed and implemented in rats, between 1989 


and 1993, and which I lately adapted and scaled up with the goal of carrying out studies 
with New and Old World non-human primates. This shift from rodents to primates 
was pivotal for our lab’s ability to follow up the original Nature Neuroscience study 
with the first demonstration of a BMI in primates, published a year later, in Nature 
(Wessberg et al., 2000). 

The next section in Volume 1, Essential Methods, contains a large list of 
manuscripts that describe in detail all the key neurophysiological, biomedical 
engineering, computational, robotics, and behavioral approaches that played a central 
role in the development of all BMIs implemented in my labs. This section is followed 
by clusters of papers that summarize our experimental studies in rodents and primates. 
Volume 2 begins with the collection of all our clinical BMI studies to date. The next 
two sections of Volume 2 are dedicated to three special BMI spinoffs that were 
originally proposed and developed at great length by our lab at Duke University. I am 
referring to the creation of sensory neuroprosthesis in rodents (our “infrared rat 
project”), and two paradigms at the edge of the field: brain-to-brain interfaces and 
brainets. In the latter section, I include a recent manuscript that reports on the first 
demonstration of simultaneous, multi-brain, multi-channel recordings based on wireless 
technology (Tseng et al., 2018). This is particularly relevant because, in the future, this 
approach will allow neurophysiologists to perform animal social studies while 
concomitantly recording the brain activity from large numbers of untethered subjects 
interacting among themselves. By the same token, this approach will allow the concept 
of “shared BMIs,” 1.e. a BMI operated by the simultaneous contributions of multiple 
individual brains engaged in the collective performance of a behavioral task. 

The final two sections of Volume 2 contain a series of opinion and review papers, 
including a very comprehensive and recent review of the entire field of BMI, including 
its historic origins, and potential future clinical applications (Lebedev and Nicolelis, 
2017). 

Revising all this material brought me a lot of great memories and rekindled some 
of the tremendous “academic battles” John Chapin and I had to fight in the beginning 
to convince some of our more recalcitrant and conservative colleagues in systems 
neuroscience that BMI was not only a legitimate experimental tool to probe the brain, 
but that, in due time, it could also become a major approach to assist and treat patients 
suffering from a variety of brain disorders that still today challenge clinicians, given the 
lack of suitable therapies to ameliorate them. Fortunately, the vast majority of these 
colleagues, given the evident success of the field and its clear potential benefits, have 
graciously changed their minds to become enthusiastic supporters of this quickly 
expanding field of inquiry. A few, sadly, still resist, calling BMI only an “applied 
science,” as if this is an offense to those of us who took the field, in twenty years, from 
its basic science origins, all the way to clinical studies that promise to unleash the full 
potential of this paradigm to millions of patients. 


Since our public demonstration of the potential impact of BMIs during the 
opening ceremony of the 2014 Soccer World Cup in Brazil, when a paraplegic patient 
was able to use a non-invasive BMI to deliver the opening kickoff of that event for a 
worldwide TV audience estimated at 1.2 billion people, the public fascination with 
BMIs, and its potential impact for the improvement of patients suffering from brain 
disorders, has skyrocketed all over the world. 

Having witnessed, many times over, the enthusiastic reaction of scientific and lay 
audiences alike all over the planet, when I show them that those humble rat studies, 
carried out in John Chapin’s Philadelphia lab in the early 1990s, allowed us, merely 
twenty years later, to offer a paraplegic young man, paralyzed from the mid-chest down, 
a way to walk again only by his own thinking, I can only say that it was all worthwhile. 

By a long shot. 


1995 


1998 


1999 


2000 


2003 


2004 


2007 


2009 


A014 


2013 


2014 


2015 


Time Table of Key Discoveries 


First demonstration of chronic, multi-site, multi-electrode recordings in 
freely behaving rodents (Nicolelis et al., Science 1995). 


First chronic, multi-site, multi-electrode recordings in non-human primates 
(Nicolelis et al., Nature Neuroscience, 1998). 


First BMI study using rodents (Chapin et al. Nature Neuroscience 1999). 


First BMI for non-human primates (Wessberg et al., Nature 2000). 
Introduction of the name “brain-machine interface” (Nicolelis Nature, 2000). 


First BMI for reaching and grasping in Rhesus Monkeys (Carmena et al. 
PLOS Biology 2003). 


First invasive multi-channel BMI in humans (Patil et al., J. Neurosurgery 
2004). 


First BMI for bipedal walking using monkey brain activity to control a 
humanoid robot (Cheng et al., Soc. Neurosci. Abstract 2007). 


A new neuroprosthesis for Parkinson’s disease based on spinal cord electrical 
stimulation (Fuentes et al., Science 2009). 


First brain-machine-brain interface for active touch exploration using a 
multichannel cortical micro-stimulation (O’Doherty et al., Nature 2009). 


First BMI for bimanual control (Ifft et al, Science Translation Medicine, 2013) 
First brain-to-brain interface (Pais-Vieira et al., Scientific Reports 2013). 

First sensory neuroprosthesis for infrared sensing (Thomson et al., Nature 
Communications 2013). 


First wireless multi-channel BMI in non-human primates (Schwarz et al., 


Nature Methods 2014). 


First brainet for non-human primates (Ramakrishnan et al., Scientific Reports 


2015). 


2016 First demonstration that chronic BMI use leads to neurological recovery in 
spinal cord injury patients (Donati et al., Scientific Reports, 2016). 


2018 First simultaneous, wireless, multi-subject brain recordings in monkeys during 
performance of a social motor task (Tseng et al., 2018). 
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TABLE 1 Results of species crosses 


Hybridization Females Males 
D. simulans—D. teissieri 
D. simulans Fla. City ° x D. teissieri Congo ¢ 51 0 
D. simulans yw @ x D. teissieri Congo 3 200 0 
D. simulans C(1)RM, yw/Y @ x D. teissieri Congo 3 o* 0) 
D. melanogaster—D. simulans 
D. melanogaster Oregon-R @ xD. simulans v° é 87 ie) 
D. melanogaster Bellows Falls 2 x D. simulans v° 3 46 0 
D. melanogaster Bellows Falls 2 x D. simulans Fla. City d 190 0 
D. melanogaster Oregon-R 2 x D. simulans vm 3 297 ce) 
D. melanogaster Napa ¢ x D, simulans w?° 3 170 a) 
D. simulans v° ° x D, melanogaster Oregon-R ¢ 83 40 
D. simulans v° @ x D. melanogaster Bellows Falls 3 176 169 
D. simulans Fla. City 2 x D. melanogaster Bellows Falls 6 62 43 
D. simulans vm @ x D. melanogaster Oregon-R 3 186 471 
D. simulans w°° 9 x D. melanogaster Napa 3 105 411 
D. melanogaster C(1)RM, ywf/YS x D. simulans v° 3 ce) 350 
D. melanogaster C(1)RM, ywf/Y2 x D. simulans vm 3 0 267 


Number of aduit hybrids appearing in species crosses. All hybrids are 
sterile. All markers are X-linked. Crosses performed at 22 °C and 18 °C (as 
results did not differ with temperature, data were pooled). In the third cross, 
numerous hybrid larvae and pupae appeared in ~30vials (the 
X teissieri/Y simuans Nybrid males survive into the pupal stage). The timing of 
hybrid lethality was determined by scoring the sex of hybrid larvae/pupae 
using the X-linked yellow marker. The reciprocal cross (D. teissieri female x 
D. simulans male) fails, apparently because of complete sexual isolation®. 
The appearance of abundant females in the D. simulans 2 x D. melanogaster 
3 crosses is not due to some rare hybrid inviability rescue mutation (H.A.O. 
and J. Coyne, unpublished results): numerous females appear when using 
all tested strains of both D. simuians and D. melanogaster (at least 7 
strains of each species tested). Tested strains include freshly caught 
wild-type flies of both species as well as older laboratory mutant and 
wild-type stocks (including flies from the United States, Australia and 
Okinawa). Although hybrid females are abundant at 18-22 °C, few appear 
at 25 °C. 

* One hybrid female appeared who was yellow*, white*, not the expected 
C(1)RM, yellow, white. This female clearly resulted from the breakdown of 
the C(1)RM chromosome (which occasionally occurs in the D. simulans stock) 
and thus carried one X chromosome from each species. 


there is little reason for believing that this remains true given 
the unusual present crossing behaviour of these species. I thus 
crossed D. melanogaster attached-X females to D. simulans 
males. As Table 1 shows, the resulting attached-X hybrid females 
remain completely inviable. Moreover, these females die during 
the same developmental period as hybrid males (late larval/early 
pupal)". 

Thus, in both hybridizations, hybrid females who are homozy- 
gous for the X from one species are as inviable as hybrid F, 
males bearing the same X. These results clearly support the 
traditional explanations of Haldane’s rule for inviability 
(although population genetic theory shows that these explana- 
tions require subtle modification (see Fig. 1)). Most important, 
these findings qualitatively differ from those obtained in 
analogous tests of Haldane’s rule for sterility? and thus show 
that the genetic basis of Haldane’s rule in Drosophila differs 
for inviability versus sterility, as several workers have specu- 
lated*-*. Haldane’s rule does not, therefore, have a single genetic 
cause, as it once appeared**”. 

Nonetheless, the genetic bases of Haldane’s rule for sterility 
versus inviability appear to differ in a logical way. In Drosophila, 
almost all mutations affecting fertility have sex-limited effects, 
whereas almost all mutations affecting viability do not". 
Similarly, the genes causing hybrid sterility clearly have sex- 
limited effects: although unbalanced hybrid females are 
homozygous for the X-linked alleles causing hybrid male steril- 
ity, they remain fertile’. Moreover, when both hybrid sexes are 
partially sterile, mapping experiments show that different loci 
affect each sex*. On the other hand, the present results suggest 
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that the genes causing hybrid inviability affect both sexes— 
females made homozygous for alleles causing hybrid male invia- 
bility are lethal. Although we cannot prove that the same loci 
cause the lethality of attached-X hybrid females and F, males, 
this conclusion is strengthened by the finding that, in both 
hybridizations, these females and males die at the same time 
during development. Moreover, in D. melanogaster-D. simulans, 
the fact that the hybrid ‘rescue mutation’ Hmr restores the 
viability of both F, hybrid males and attached-X hybrid 
females'’ strongly suggests that these hybrids suffer from the 
same developmental problem. 

In conclusion, the behaviour of the factors causing reproduc- 
tive isolation between species is strikingly similar to that of 
alleles affecting fitness within species: factors affecting fertility 
have sex-limited effects, but those affecting viability apparently 
do not. This parallel provides the clearest evidence to date that 
speciation involves the divergence of ‘ordinary’ genes with nor- 
mal functions within species'*, as proposed by the founders of 
the modern synthesis. 
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Induction of immediate 
spatiotemporal changes in thalamic 
networks by peripheral block of 
ascending cutaneous information 
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PERIPHERAL sensory deprivation induces reorganization within 
the somatosensory cortex of adult animals'*. Although most 
studies have focused on the somatosensory cortex’, changes at 
subcortical levels (for example the thalamus) could also play a 
fundamental role in sensory plasticity’ ''. To investigate this, we 
made chronic simultaneous recordings of large numbers of single 
neurons across the ventral posterior medial thalamus (VPM) in 
adult rats. This allowed a continuous and quantitative evaluation 
of the receptive fields of the same sample of single VPM neurons 
per animal, before and after sensory deprivation. Local anaesthesia 
in the face induced an immediate and reversible reorganization 
of a large portion of the VPM map. This differentially affected 
the short latency (4-6 ms) responses (SLRs) and long latency 
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FIG. 1 Local anaesthesia of the rostral face alters the sensory responses 
of three simultaneously recorded neurons in the VPM of a pentobarbital 
anaesthetized rat. For each cell, cumulative frequency histograms (CFHs; 
above) and post-stimulus time histograms (PSTHs; below) show the magni- 
tude and timing of different latency responses of these neurons to 360 
single stimulations of different whiskers (E5, E1 and C2) before (Aa-—Ca) 
and after (Ab-Cb) subcutaneous lidocaine injection in the maxillary gum. 
Response magnitudes, in average instantaneous firing rate per bin (in spikes 
per s) are shown on the vertical axes of PSTHS. The scales are different 
for each PSTH to highlight the effects. The vertical dimension in CFHs is 
used to plot the deviation of the cumulative firing above or below the average 
firing rate. Scales on the vertical axes of CFHs show negative log P-values 
(for example at the first scale mark P=0.1, at the second, P=0.01 and so 
on). These show the probability that the overall distribution of cumulative 
frequencies differs from arandom distribution, as computed using a one-way 
Kolmogorov-Smirnov test. Thus, the highly significant response in A 
(P<107®) became insignificant (P > 0.1) after the lidocaine injection. Horiz- 
ontal axis: pre- and post-stimulus time in s; whisker deflection up at 0, 
down at 0.1 s; bins=1 ms, scale marks=5, 25 and 100 ms. 

METHODS. Experiments were done on six adult Long-Evans (hooded) rats. 
One week before experimentation bundles of 8-16 teflon-coated stainless 
steel 25 or 50 ym tip microwires were surgically implanted in the VPM 
thalamic nucleus. Up to 23 single VPM neuronal signals were simultaneously 
amplified, filtered and discriminated using an apparatus from Spectrum 
Scientific (Dallas, TX). Each single extracellular spike waveform was time- 
voltage window discriminated, and statistically validated using principal 
components analysis. This ensured that the same set of single neurons 
was recorded throughout each experiment. Cutaneous sites were stimulated 
at a rate of 1 Hz, using a computer-controlled vibromechanical probe deliver- 
ing 100 ms duration step pulse displacements (whiskers were moved 3°, 
furry skin was depressed 0.5mm). For each experiment, 11-20 single 
whiskers or skin sites were independently stimulated in a random sequence. 
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(15-25 ms) responses (LLRs) of single VPM neurons. The SLRs 
and LLRs normally define spatiotemporally complex receptive 
fields in the VPM'*. Here we report that 73% of single neurons 
whose original receptive fields included the anaesthetized zone 
showed immediate unmasking of SLRs in response to stimulation 
of adjacent cutaneous regions, and/or loss of SLRs with preserva- 
tion or enhancement of LLRs in response to stimulation of regions 
just surrounding the anaesthetized zone. This thalamic reorganiz- 
ation demonstrates that peripheral sensory deprivation may induce 
immediate plastic changes at multiple levels of the somatosensory 
system. Further, its spatiotemporally complex character suggests 
a disruption of the normal dynamic equilibrium between multiple 
ascending and descending influences on the VPM. 

To investigate whether peripheral deprivation produces 
immediate reorganization of sensory maps in subcortical struc- 
tures such as the thalamus, we developed neurophysiological 
techniques allowing chronic simultaneous recordings of large 
numbers of single neurons distributed across topographic rep- 
resentations in sensory nuclei. This approach overcame many 
limitations of the classic mapping technique because it allowed: 
(1) the same neurons to be recorded before and after sensory 
deprivation, (2) a precise quantitative definition of their spatio- 
temporal response properties, and (3) a simultaneous sampling 
of short- and long-term neuronal changes occurring across the 
whole sensory representation embedded in the nucleus. This 
new approach was used to demonstrate that the map of the face 
in the VPM thalamus of adult rats is immediately reorganized 
following reversible peripheral sensory deprivation induced by 
small (0.01-0.04 ml) subcutaneous injections of lidocaine (1%). 
This reorganization was manifested by marked changes in the 
spatiotemporal response properties of single VPM neurons to 
stimulation of facial locations within the fully and partially 
anaesthetized zones, and also their immediate surrounds. 

Figure 1 illustrates typical spatiotemporal modifications 
observed within an ensemble of single thalamic neurons follow- 
ing a small (0.04 ml) injection of lidocaine in the gum just behind 
the maxillary incisors in a pentobarbital anaesthetized (50 mg 
kg”! intraperitoneally) adult rat. Before this injection, most of 
the total 11 neurons that were simultaneously recorded in this 
rat responded robustly to computer controlled vibromechanical 
stimulation of up to 16 whiskers across the face, including 
whisker E5 (as in Fig. 1Aa). After the injection, these responses 
to E5 were completely blocked (Fig. 1Ab), thus defining this 
rostral whisker as part of the zone anaesthetized by the lidocaine 
spread. By contrast, stimulation of more caudally placed whis- 
kers (for example, whisker E1) yielded strong short latency 
(5-10 ms) responses in many neurons that had previously been 
unresponsive at that latency (Fig. 1 Ba, b). Such complementary 
changes in response magnitudes produced a spatial shift of 
single neuronal receptive fields away from the anaesthetized 
zone towards surrounding unanaesthetized regions. 

Beyond these changes in the spatial domain, the focal anaest- 
hesia also altered the normal latency distribution of sensory 
responses of 10 neurons within the ensemble. For example, the 
neuron in Fig. 1Ca normally responded at both short (5-10 ms) 
and long (15-25 ms) latencies to stimulation of whisker C2. 
Lidocaine selectively blocked the short latency response, thus 
shifting the temporal profile of the post-stimulus time histograms 
(see Fig. 1Cb). 

The statistical significance of the sensory-evoked responses 
in these neurons was routinely assessed using: (1) the Kol- 
mogorov-Smirnov (KS) test, which estimates the deviation from 
randomness of the cumulative frequency distribution, and (2) 
analysis of variance (ANOVA), which determines whether 
sensory-evoked responses in specific time epochs are sig- 
nificantly different from spontaneous discharge. All of the effects 
reported here were found to be highly significant (P<0.01). 
The deprivation-induced changes occurred immediately (within 
about 3 minutes) after the lidocaine injection and lasted 4-6 
hours. In all cases in which the neurons were re-recorded 12-24 
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FIG. 2 Spatiotemporal receptive fields of VPM 
neurons are altered by lidocaine anaesthesia 
of the rostral face. The colour-coded three- 
dimensional surfaces here define the spatiotem- 
poral receptive fields of two VPM neurons before 
(Aa and Ba) and after (Ab and Bb) a lidocaine 
injection in the maxillary gum of an adult rat. These 
were constructed by sequentially stimulating 11 
whiskers 360 times apiece, and measuring 
average instantaneous evoked firing rate during 
each of eight different post-stimulus time epochs 
(shown on the left horizontal axis, ranging from O 
to 35ms). The responses for each stimulated 
whisker were rank ordered according to their rela- 
tive caudal-to-rostral position in the whisker pad 
(ordered from 1-11 as follows along the right 
horizontal axis: B1, £1, E2, C1, C2, D2, E3, C3, E4, 
E5 and E6. Response magnitudes are shown in Hz 
(spikes per s) on the vertical axis. The range of 
response magnitudes was coded by eight colours, 
in which red represents the greatest firing rate 
and dark blue the lowest. Ail colour codes were 
automatically scaled according to the vertical 
range, and thus vary from graph to graph. Actual 
data points are represented by white open circles 
whose distances from the spline smoothed sur- 
face are indicated by vertical dotted lines. All of 
the major topographical features shown here were 
shown to be significant (P<0.01) using the 
ANOVA and KS tests. 


hours after the lidocaine injections their receptive fields were 
found to be similar to pre-lidocaine controls. 

The major conclusion to be derived from the above analyses 
of PSTHs is that peripheral sensory deprivation disrupts both 
spatial and temporal features of thalamic neuronal responses. 
To visualize how the responses seen in PSTHs (as in Fig. 1) 
contribute to the definition of single cell receptive fields, we 
used three-dimensional computer graphics to construct ‘spatio- 
temporal receptive fields’, which depict neuronal response mag- 
nitude (in spikes per s) as a function of stimulus position and 
post-stimulus time. For this, quantitative measurements from 
the PSTHs were used to generate a three-dimensional surface, 
which was smoothed using a spline algorithm based on a moving 
third-order polynomial function. To reduce the impact of statis- 
tical outliers, a low ‘stiffness’ coefficient (0.15) was used. 

Figure 2 shows spatiotemporal receptive fields of two typical 
neurons before (Fig. 24a, Ba) and after (Fig. 2Ab, Bb,) the 
same lidocaine injection as in Fig. 1. These quantitative rep- 
resentations revealed that receptive fields in the VPM are much 
larger and spatiotemporally complex'* than reported!*!>. 
Lidocaine induced two general categories of receptive field 
reorganization in the VPM: (1) temporal shifts (as in Fig. 2Aa, b) 
and (2) spatial shifts (as in Fig. 2Ba, b). Figure 2Aa depicts a 
receptive field that covers a wide area of the whisker pad during 
both short and long latency post-stimulus time epochs. After 
the lidocaine injection, this receptive field changed dramatically 
(Fig. 2Ab). At short latencies, the centre of the receptive field 
disappeared, while its surround was enhanced at both edges. 
By contrast, the long latency peak remained almost intact, 
though somewhat prolonged. Thus, the main effect of the depri- 
vation was to shift temporally the main response of this neuron 
from short to long latency epochs. 

By contrast, Fig. 2 Ba, b shows an example of a predominantly 
spatial shift. Lidocaine-induced sensory deprivation produced 
a marked spatial shift in the short latency receptive field peak 
of this neuron from its original location in the rostral whiskers 
(C3 and E3-ES) to the caudal whiskers (B1, C1 and E1-E2). 
In addition, two weak long latency response peaks emerged. 
Similar patterns of reorganization were observed in each of eight 
equivalent experiments, involving chronic recordings from a 
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FIG. 3 Population receptive fields of VPM neuronal ensembles are altered 
by lidocaine anaesthesia in the rostral face. Each colour-coded three- 
dimensional surface here represents the sensory-evoked responses of an 
ensemble of 14 simultaneously recorded VPM neurons to stimulation of 
each of 11 whiskers (same as in Fig. 2) within each of three post-stimulus 
time epochs (5-10, 10-15 and 25-35 ms). Aa-Ca show the population 
receptive fields recorded during control. Ab-Cb show the same population 
receptive fields following the lidocaine injection. The vertical axis, right 
horizontal axis, and the coiour coding of the three-dimensional surfaces are 
the same as in Fig. 2. The 11 neurons that comprise the left horizontal axis 
were rank ordered according to the relative position of their receptive field 
centres and their overall sensory responsiveness. 
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total of 75 VPM neurons in six rats. Of the 75 neurons, 55 had 
receptive fields within the anaesthetized zone and/or its immedi- 
ate surround. Of these, 30 cells (55%) exhibited the temporal 
shifts (category 1, as in Fig. 2Aa, b), and 27 cells (49%) exhibited 
the spatial shifts (category 2, as in Fig. 2Ba, b). These effects 
were partially overlapped, such that both were observed in 17 
cells (31%). Finally, 11 cells did not change, and 4 cells were 
completely anaesthetized. 

The next step was to determine how the spatiotemporal com- 
plexity of the lidocaine effects on the receptive fields of single 
neurons was translated into changes in the sensory map embed- 
ded in the neuronal population. Three-dimensional graphics 
were used to construct population receptive fields, which depict, 
within given post-stimulus time epochs, the magnitudes of 
sensory responses across the population of recorded neurons as 
a function of stimulus location. The population receptive fields 
in Fig. 3Aa-Ca show how the spatiotemporal receptive fields 
of 11 simultaneously recorded single neurons were integrated 
into a sensory map which dynamically changes over post- 
stimulus time. The lidocaine produced the alterations shown in 
Fig. 3Ab-Cb by creating an anaesthetized zone in which short 
latency responses to stimulation of whiskers C3 and E3-E5 
(7-10 in Fig. 3Ab and Bb) were blocked. This induced a spatial 
shift of the short latency responses toward ‘far surround’ whis- 
kers Bi, El, and E6 (1,2 and 11 in Fig. 3Bb), distal to the 
anaesthetized zone. Furthermore, long latency responses tended 
to remain, or newly appear, in ‘near surround’ whiskers E2, C1 
and 2, and D2 (3-6 in Fig. 3Cb), more proximal to the anaesthet- 
ized zone. The net effect was to increase the relative strength of 
the longer latency responses which partially ‘filled in’ the area 
vacated by the short latency responses, conceivably providing 
a ‘seed’ for longer term reorganization. 

These results demonstrate that local peripheral sensory depri- 
vation induces, within minutes, a marked reorganization of the 
somatosensory map in the VPM thalamus. The purely spatial 
aspects of these receptive field shifts are reminiscent of the 
spatial reorganizations of the sensory map in the somatosensory 
cortex which were defined using classic mapping studies'->. 
Moreover, they are quite similar to the immediate unmasking 
of cortical neuronal responsiveness observed after similar 
sensory deprivation models*®. The results here, therefore, show 
that the short-term plastic reorganization in the somatosensory 
system is not restricted to the cerebral cortex, and provide direct 
quantitative evidence for previous suggestions that the thalamic 
map may be reorganized after different types of sensory depriva- 
tion” '''*?5_ Thus, the long-term reorganizations observed in 
the cortex could derive at least partially from changes at lower 
levels such as the thalamus. 

Another important issue raised by the quantitative approach 
used here is that thalamic receptive fields, which normally 
exhibit complex spatiotemporal dynamics'*, are altered by 
peripheral deafferentation. This spatiotemporal complexity 
might be attributed to the asynchronous convergence of multiple 
feedforward'*-'° and feedback’ excitatory inputs to the VPM, 
which could be the sources of the different latency response 
components observed in these neurons. This is supported by 
our observations that short and long latency responses in the 
VPM have different receptive fields'*, and further that lidocaine 
affects them selectively. The lidocaine-induced unmasking of 
these various components could result from changes in the tonic 
activity of inhibitory neurons acting on the VPM or trigeminal 
nuclei. As a hypothesis, therefore, we suggest that sensory depri- 
vation disrupts the normal dynamic state of equilibrium between 
excitation and inhibition within the network that comprises the 
entire somatosensory system, producing reorganizations at 
multiple levels of this pathway. As such, the population receptive 
fields in Fig. 3Ab-Cb could be considered as snapshots of the 
new dynamic equilibrium states of the somatosensory network 
as viewed at the thalamic level. This theoretical framework 
for a network-based model of sensory deprivation-induced 
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plasticity might reasonably be applied to other systems of the 
brain. 
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THE neurotransmitter dopamine acts through various receptor 
subtypes that are largely associated with enhancement or inhibition 
of adenylyl cyclases'”. These dopamine-sensitive adenylyl cyclases 
are highly concentrated in the corpus striatum and associated 
limbic structures of the brain, where their levels exceed by orders 
of magnitude** those in other areas of the brain. Here we use in 
situ hybridization to show that messenger RNA for three of these 
adenylyl cyclases*’ is not found in the corpus striatum. We have 
isolated and expressed a complementary DNA encoding new adeny- 
lyl cyclase whose selective concentration in the corpus striatum 
indicates that it may be responsible for the synaptic actions of 
dopamine. 

Binding of [7H] forskolin to adenylyl cyclase is highly concen- 
trated in the corpus striatum (ST), the related limbic areas, and 
the substantia nigra to which ST fibres project** (Fig. 1). For- 
skolin can interact with glucose transporters®, but cytochalasin 
B, which potently competes with forskolin at the glucose trans- 
porter, does not change the pattern of [*H]forskolin auto- 
radiography in the brain (data not shown). In situ hybridiz- 
ation reveals no evidence for concentration in ST of mRNA for 
type I, type II or type III adenylyl cyclase. The type I enzyme 
is most concentrated in granule cells of the cerebellum and the 
dentate gyrus of the hippocampus, as found previously’. Type 
II is less abundant than type I, with some concentration in 
granule cells of the cerebellum, CA-1 and CA-4 of the hippocam- 
pus, as well as in granule cells of the dentate gyrus. Type III 
adenylyl cyclase is reported to be selectively localized to olfac- 
tory neurons’, but we observe specific mRNA signals for type 
UI that are more homogeneously distributed throughout the 
brain than type I or type II adenylyl cyclases, with some con- 
centration in the granule cells of the cerebellum and throughout 
the hippocampus. No concentration in the ST is apparent for 
types I, I] or Il] enzymes; type IV is found in rat brain in very 
small amounts'®. Perhaps because of this scarcity, we could not 
detect type IV mRNA by in situ hybridization (data not shown). 
Sense-strand controls showed that nonspecific hybridization was 
low in all experiments (data not shown). 
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Sensorimotor Encoding by Synchronous Neural 
Ensemble Activity at Multiple Levels of the 
Somatosensory System 
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John K. Chapin 


Neural ensemble processing of sensorimotor information during behavior was investi- 
gated by simultaneously recording up to 48 single neurons at multiple relays of the rat 
trigeminal somatosensory system. Cortical, thalamic, and brainstem neurons exhibited 
widespread 7- to 12-hertz synchronous oscillations, which began during attentive im- 
mobility and reliably predicted the imminent onset of rhythmic whisker twitching. Each 
oscillatory cycle began as a traveling wave of neural activity in the cortex that then spread 
to the thalamus. Just before the onset of rhythmic whisker twitching, the oscillations 
spread to the spinal trigeminal brainstem complex. Thereafter, the oscillations at all levels 
were synchronous with whisker protraction. Neural structures manifesting these rhythms 
also exhibited distributed spatiotemporal patterns of neuronal ensemble activity in re- 
sponse to tactile stimulation. Thus, multilevel synchronous activity in this system may 
encode not only sensory information but also the onset and temporal domain of tactile 
exploratory movements. 


The processing of somatosensory informa- ed by synchronous oscillations or even cha- 
tion in the mammalian brain involves the — otic behavior (4), that could be computa- 
transmission of neural activity from the tionally useful (5). The existence of this 
skin to the neocortex by way of parallel network raises the question of whether 


pathways that ascend through a hierarchi- —_ large-scale coordinated activity of neural 
cal sequence of neural structures (1). Like | ensembles involving multiple levels of a 
those in other sensory systems, these as- sensory system (that is, brainstem, thala- 
cending pathways are far outnumbered by — mus, and cortex) can play a fundamental 
corticofugal descending projections (2), role in the coding of sensory information. 
which can act at multiple subcortical levels | To address this issue, we simultaneously 
to modify the processing of sensory infor- recorded up to 48 single neurons, distribut- 
mation (3). These connections define a — ed across up to five distinct processing re- 
recurrent network that is theoretically ca- __ lays of the trigeminal somatosensory system, 
pable of generating complex emergent dy- _ in freely behaving rats (6). The rat trigem- 
namic patterns of neural activity, manifest- inal system is a multilevel, recurrently in- 


terconnected neuronal network, specialized 
M. A. L. Nicolelis, R. C. S. Lin, J. K. Chapin, Department for processing complex patterns of tactile 
of Anatomy and Neurobiology, Medical College of Penn- —_ stimuli generated by the repetitive contacts 
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ditions (8), little is known about informa- 
tion processing at the neural ensemble level 
during exploratory tactile behaviors. 

The results described here are based on 
the long-term behavior of 424 single neu- 
rons chronically recorded in 11 animals. 
Sampled structures included the trigeminal 
ganglion (Vg, 17 cells), the principal (PrV, 
52 cells) and spinal (SpV, 39 cells) trigem- 
inal brainstem nuclei, the ventral posterior 
medial (VPM, 186 cells) and posterior me- 
dial (POm, 19 cells) nuclei of the thalamus, 
the primary somatosensory (SI, 95 cells) 
cortex, and the primary motor (MI, 
cells) cortex. Pairwise cross correlation 
analysis was initially used to characterize 
the coordinated activity of simultaneously 
recorded neurons across the trigeminal sys- 
tem. Cross correlograms averaged around 
the spiking of a single representative VPM 
neuron (Fig. 1A) in each animal demon- 
strated the existence of synchronous oscil- 
latory discharges (7 to 12 Hz) in 98% of the 
ipsilateral VPM neurons, 92% of the ipsi- 
lateral SI cortical neurons, and 49% of the 
contralateral SpV neurons recorded in this 
study. During episodes of synchronous fir- 
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ing, thalamic neurons tended to phase-lag 
(follow) cortical neurons (Fig. 1A) by time 
periods ranging from 0 to 20 ms (mean + 


SD, 9.19 + 4.67 ms; n = 84 SI-VPM pairs in 


six animals). The regular progression of 


these lags across evenly spaced electrode 
arrays implanted in the SI cortex (for an 
example see the bottom third of Fig. 1A) 


was compatible with a traveling wave of 


neuronal activity crossing this cortical re- 
gion. Similar synchronous oscillations were 
also observed among neurons in the whisker 
representation of the rat MI cortex, and, like 
the SI cortical neurons, these neurons 
phase-led neurons in the VPM thalamus. 
The simultaneously recorded neurons in the 
brainstem SpV nucleus were also synchro- 
nized with this oscillation, and these neu- 
rons phase-led neurons in the VPM thala- 
mus by as much as 40 ms (Fig. 1A). By 
contrast, neurons within the VPM and POm 
thalamus exhibited remarkably synchronous 
firing with minimal phase lags (<3 ms). 
This high degree of synchrony was observed 
bilaterally in the thalamus, implying that all 
of these thalamic structures were simulta- 
neously entrained into the oscillations. 
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The possibility that these oscillations 
were caused by rhythmic sensory feedback 
resulting from active whisker movements 
(WMs) was ruled out by three observations. 
First, as detailed below, they began well 
before the onset of WMs. Second, the same 
pattern of oscillations was clearly present in 
adult animals subjected to complete unilat- 
eral whisker removal or facial nerve section 
in early development (9). Third, the oscil- 
lations were not observed in either the first- 
order (Vg) or the primary second-order 
(PrV) sensory relays in this pathway. 

To further examine the neurophysio- 
logical properties of the simultaneously 
recorded multilevel neural ensembles, we 
developed analytical techniques for repre- 
sentation of neuronal population func- 
tions. We used principal components 
analysis (PCA) to construct continuous 
representations of the activity of neuronal 
populations by weighted summation of the 
time-integrated activity of all neurons 
(10). We observed that reconstructions of 
the first principal component (PC1), de- 
tived from combined cortical, thalamic, 
and brainstem neural ensemble activity, 
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Fig. 1. Spontaneous 7- to 12-Hz oscillations at multiple relays of the trigeminal 
somatosensory system. (A) Cross correlograms (CCs), calculated for 16 out of 
a total 48 simultaneously recorded neurons, reveal synchronous 7- to 12-Hz 
oscillations at three levels of the trigeminal pathway (SpV, 4 neurons; VPM, 6 
neurons; and Sl, 6 neurons). All CCs centered around the spiking of one 
reference VPM neuron (autocorrelation shown at arrow). Numbers on top of 
CCs indicate the time interval (in milliseconds) by which each SpvV or SI neu- 
ron phase-leads the reference VPM neuron. All horizontal axes indicate 
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pre- and post-VPM spike times (in seconds). VPM spiking occurred at 0.0 s, 
as indicated by the vertical dashed lines. Vertical axes of CCs indicate equiv- 
alent firing rates per bin (bins: 1-ms duration). (B) Continuous strip chart 
showing 43 single-unit rasters and reconstructed PC1 (bottom) over the same 
5.0-s time period in an awake rat. The rasters show spiking activity of simul- 
taneously recorded neurons in four system relays; PrV, SpV, VPM, and SI. Not 
shown are the five neurons recorded in Vg, which were inactive over this 
period. The vertical axis depicts the weighted neuronal population firing rate. 


more reliably identified the existence and 
time course of oscillatory episodes than 
did the raster displays of single neuronal 
spiking (Fig. 1B). This difference was due 
to the marked variability in spiking pat- 
terns (timing and number of spikes) of 
single neurons over different oscillatory 
cycles. Indeed, only a fraction of the pop- 
ulation fired during a particular cycle. 
Thus, oscillations revealed in PC1 were 
not completely defined by any one neuron 
but instead drew small amounts of vari- 
ance from each of a large population of 
neurons. Because PC1 normally explains 
the most global sources of covariance be- 
tween variables, it is significant that it 


tended to be dominated by these wide- 
spread synchronous oscillations (11). 
Spectrum analysis of the time series rep- 
resenting PCI for the multilevel data dem- 
onstrated that the most important power 
contribution of the oscillations was in the 
frequency band from 7 to 12 Hz, with a 
secondary peak at 19 to 21 Hz. The oscil- 
lations appeared in episodes ranging from 
0.8 to 38.2 s (mean + SE, 3.96 + 0.3 s;n = 
196 episodes) and were closely associated 
with preparation for movement (Fig. 2). 
Although they always began during com- 
plete immobility (Fig. 2A), in 79.1% (155 
of 196, from five animals) of the recorded 
episodes, these oscillations predicted with a 
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Fig. 2. Association between oscillations B 
and WT behavior. (A) Four strip charts 
depicting 80.0 s of continuous activity of —— 
PC1, reconstructed from the activity of — 
48 neurons recorded at five levels ofthe ===— 


trigeminal pathway. This PC1 shows All 
several episodes of synchronous 7- to 
12-Hz_ oscillations. | Frame-by-frame 
analysis of video tape synchronized to 
this experiment showed that these oscil- 
lations were associated with a distinctive 
succession of behaviors, beginning with 
rest (R), followed by low-amplitude WT, 
and ending with high-amplitude explor- sl 
atory WMs, which were often associated 
with HMs. For example, after a period of 
intermittent movement, the animal as- 
sumed an immobile attentive posture (R) 
at 18.5 s, which continued until the on- 
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set of oscillations (7 to 12 Hz) at 20.5 s. About 500 ms later, a sequence of low-amplitude WT (21.2 to 
25.1 s) began. This WT was then followed by WM and HM, which terminated the oscillations. Vertical 
arrows mark the onset of various behaviors. (B) Phase synchronization between whisker protraction and 
7- to 12-Hz oscillations at multiple levels of the trigeminal system. A raster plot (top) and PC 1s (for different 
levels) are shown centered around the onsets of 64 episodes of whisker protraction during WT. Each 
black horizontal line in the raster defines the time period between protraction onset (vertical dotted line at 
0.0 s) and subsequent retraction onset. The horizontal dotted line above the raster plot depicts the 
average duration (49 ms) of whisker protraction during continuous WT. The PC1s calculated for all levels 
together (topmost PC1 plot) or for each individual level (SpV, VPM, and Sl) are averaged around the onset 
of whisker protraction. All PC1s are tightly phase-locked to whisker protraction, and the SpV, SI, and VPM 
activity peaks occur during the early, middle, and late phases of protraction, respectively. The vertical axis 


depicts the weighted neuronal population activity. 
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high degree of statistical confidence (P < 
10°) the imminent onset of low-amplitude 
whisker-twitching (WT) movements (12). 
Even though WT began on average 581 + 
45.2 ms after the oscillation onset, the dis- 
tribution of these latencies was markedly 
skewed, such that 62.5% of all WT episodes 
began during the first 500 ms. The frequen- 
cy range of WT (7) closely matched the 
frequency of neural oscillations in the tri- 
geminal system (7 to 12 Hz). In fact, after 
the onset of WT, these oscillations assumed 
a phase-locked relation with this move- 
ment: Oscillatory peaks in the SpV were 
synchronous with the early phase of whisker 
protraction, whereas peaks in the SI and 
VPM were synchronous with the middle 
and late phases of whisker protraction, re- 
spectively (Fig. 2B). Subsequent onset of 
high-amplitude rhythmic exploratory WMs, 
which were often combined with head 
movements or locomotion, always induced 
a cessation of these oscillations. In the ab- 
sence of WT, head movements (HMs) 
alone were never preceded by oscillations 
(see HMs starting at 13.7 s in Fig. 2A). 
Independent reconstruction of principal 
components (PCs) at different levels of the 
trigeminal pathway revealed that in 79.8% 
(138 of 173, n = five animals) of the oscil- 
latory sequences, rhythmic activity clearly 
began in the SI cortex (Fig. 3A, point a) 
and progressed to the VPM thalamus (Fig. 
3A, point b). Neurons in the SpV tended to 
be recruited last, exhibiting clear rhythmic 
activity 1.53 + 0.07 oscillatory cycles 
(~100 ms per cycle) before the onset of 
WT (Fig. 3A, dashed line), as averaged 
across 114 episodes in which SpV neurons 
were recorded (Fig. 3A, point c). Although 
the SpV neurons were the last to be recruit- 
ed into rhythmic activity, after the onset of 
WT they phase-led both the SI and VPM 
neurons (Figs. 2B and 3B). This observation 
argues against a simple pacemaker mecha- 
nism for these oscillations. 

The classification of this oscillatory phe- 
nomenon remains elusive (13). Its associa- 
tion with attentive behaviors resembles 
that of “mu” rhythms recorded in humans 
(14) and the “sensorimotor” rhythms de- 
scribed in cats and primates (15). Our data 
suggest that similar oscillatory activity is 
present at multiple processing levels of the 
rat somatosensory system and that it can be 
definitively related to a specific type of 
rhythmic movement (WT). Differences in 
the frequency of this rhythm across species 
might therefore be related to different time 
domains of movements for which it is pre- 
paratory. Our hypothesis is that the 7- to 
12-Hz frequency component of the oscilla- 
tions defines an “internally generated rep- 
resentation” of the sensorimotor temporal 
domain of exploratory movements, espe- 
cially WT, which is disseminated across 
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much of the somatosensory pathway before 
the onset of the movement. 


/?~» We propose that these synchronous os- 


-illations provide a phasic sensory facilita- 
tion function timed to coincide with whis- 
ker protraction, which is known to be the 
period in which active sampling of tactile 
information occurs in rats (7). In this mod- 
el, the observed 19- to 21-Hz frequency 
component might constitute a representa- 
tion of the timing of alternating subphases 
of WM—for example, protraction and re- 
traction (Fig. 2B). Further support for this 
model is provided by our observation that, 
depending on the oscillatory phase at which 
the incoming sensory volley arrives, these 
rhythms can modify the response to the 
stimulus (Fig. 3C). 

The strong correlation of these sensori- 
motor rhythms with immobile attention 
leading to WT clearly distinguishes these 
thythms from other brain rhythms such as 
spindling (16). To further clarify this dis- 
tinction, we compared the sensorimotor 
rhythms with spindling activity in the same 
animals and found that there were several 
differences between these two rhythms. 
First, spindles occurred during drowsiness 


(as opposed to attentive resting) and during 
the initial stages of sleep, in the complete 
absence of WMs. Second, oscillatory se- 
quences during spindling occurred in much 
shorter (0.4 to 1.2 s) and stereotyped epi- 
sodes than did the sensorimotor rhythms 
described during awake immobility. Third, 
unlike the relatively constant amplitude of 
the sensorimotor oscillations, the spindles 
exhibited very characteristic waxing and 
waning wave patterns. Finally, spindle epi- 
sodes recurred every 5 to 10 s, whereas the 
occurrence of sensorimotor rhythms was 
linked to behavior changes and was there- 
fore more random. Thus, although spindles 
involved similar frequencies (7 to 14 Hz) 
and were observed in the same thalamocor- 
tical circuits, they clearly subserve a very 
different functional role from the sensori- 
motor rhythms described here. 

To further define the dynamic nature 
of sensory processing among the multiple 
processing levels of the rat trigeminal sys- 
tem during awake immobility, we carried 
out repetitive stimulation of single whis- 
kers in the same animals. Population post- 
stimulus time histograms (PPSTHs) were 
used to represent the simultaneously re- 
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corded sensory responses of the same en- 
semble of neurons to repeated stimulation 
of a given whisker (Fig. 4). This analysis 
revealed that Vg and PrV neurons (1 to 13 
in the vertical scale of Fig. 4) responded 
well to the stimulation of a single whisker 
(B4, Fig. 4B) but exhibited little or no 
response to stimulation of other sites (such 
as whisker B2, Fig. 4A, and fur rostral to 
B4, Fig. 4C). In contrast, most of the SpV 
(14 to 22), VPM (23 to 31), and SI (32 to 
47) neurons displayed statistically signifi- 
cant (P < 0.01) responses to all three 
stimulation sites. In the ascending pro- 
gression from V¢ to SI, the latencies to the 
initial response increased from 2 to 8 ms. 
In addition, secondary long-latency re- 
sponses were evident through much of this 
system, appearing first in a few PrV neu- 
rons and becoming more prominent and 
longer in latency at higher levels. For 
instance, thalamic cells exhibited second- 
ary long-latency responses up to 35 ms, 
and, in the SI cortex, secondary long- 
latency responses ranged from 20 to 70 ms 
after the stimulus. As previously demon- 
strated for the VPM neurons (17), these 
results showed that sensory maps at most 
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Fig. 3. (A) Oscillations appear first in the SI cortex. Four strip charts show four Cc 
PCs derived independently from simultaneously recorded neurons in specific 500 


nuclei: SpV (11 neurons), VPM (8 neurons), and SI (16 neurons). For SI, both 
PC1 and PC2 are shown. Oscillatory peaks begin in the SI neurons (a), 
spread to the VPM neurons (b) after one to two cycles, and finally reach the 
SpV neurons (c) after five to six cycles. (B) Strip charts depicting the PC1s for 
the SpV neurons (dark line) and the VPM neurons (dotted line) show that in 
the VPM neurons, oscillations are continuous for more than four cycles 
before recruitment of the SpV neurons into the oscillations. Nevertheless, the 
SpV neurons clearly phase-lead the VPM neurons thereafter. The recruitment 
of the SpV neurons coincides with the onset of WT movements (vertical 
arrow). (C) Nonlinear interactions between 7- to 12-Hz oscillations and sin- 
gle-whisker stimulations are represented by a series of four strip charts, each 
depicting 2.5-s sequences of PC1 derived from 23 VPM neurons. Single- 
(pmibisker stimulation trials (vertical bars, circled numbers) superimposed on 
1e PC1 record show that sensory stimulation can terminate ongoing oscil- 
lations (1), modify the phase of the oscillations (2, 5, and 6), produce an 
isolated sensory response (9), initiate oscillations (4 and 8), have no effect (7 
and 10), and augment the oscillation peak (9). The vertical axis depicts the 
weighted neuronal population firing rate. 
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levels of the trigeminal system (SpV, 
VPM, POm, and SI) were defined by com- 
plex spatiotemporal patterns of neural en- 
semble responses. Therefore, although the 
Vg and PrV structures contained relatively 
simple topographic sensory representa- 
tions and did not exhibit premovement 
oscillations, structures that manifested 
synchronous oscillations (SpV, VPM, 
POm, and SI) exhibited distributed (18) 
and dynamic somatosensory maps. 

The complex spatiotemporal structure of 
these maps may result from asynchronous 
interactions between feedforward and feed- 
back projections at each level of this sys- 
tem. As a consequence, the coding of sen- 
sory information in most cortical and sub- 
cortical relays of the trigeminal pathways 
occurs at the ensemble rather than at the 
single-unit level and involves both spatial 
and temporal domains. According to this 
scenario, topographic maps defined in anes- 
thetized animals represent only a first ap- 
proximation of the system’s functional or- 
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ganization. In the awake condition, how- 
ever, spatiotemporal complexity substitutes 
for topography as the main strategy for the 
coding of sensory information. 

These results indicate that the func- 
tional organization of the rat somatosen- 
sory system is fundamentally linked to the 
coordinated activity of large ensembles of 
neurons, distributed through multiple lev- 
els of the brain. Dynamic patterns of neu- 
ral ensemble activity in this sensory sys- 
tem were found not only to code tactile 
stimulus attributes but also to anticipate 
the occurrence of stereotyped WT behav- 
iors associated with active tactile explora- 
tion of the surrounding environment: Be- 
cause these oscillations appear to mimic a 
motor output function, they are consistent 
with earlier demonstrations in rats, cats, 
and monkeys that the transmission of sen- 
sory information through different levels 
of the somatosensory system is strongly 
modulated as a function of the phase of 
active movement (19). 
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Fig. 4. Color-coded PPSTHs depicting the spatiotemporal sensory response patterns of 47 neurons 
simultaneously recorded across five levels of the trigeminal system in an awake rat. The absolute neuronal 
firing intensities are displayed through a seven-color contour plot [dark red, highest firing rate (up to 120 
Hz); dark blue, lowest firing rate (as low as 0 Hz)]. Neurons that are included in this multilevel ensemble are 
organized in the vertical axis (Vg, 1 to 4; PrV, 5 to 13; SpV, 14 to 22; VPM, 23 to 31; and Sl, 32 to 47). 
Poststimulus time is represented in 5-ms time bins numbered from 1 to 10 (0 to 50 ms). The graphs were 
produced by calculating the intensities of spike discharge of each neuron within the simultaneously 
recorded ensemble, rank-ordering the neurons into rows according to receptive field location, and then 
plotting the data into a color-coded contour plot image. Each PPSTH depicts the spatial extent and timing 
of the spread of sensory responses through the nuclei in which neuronal ensembles were simultaneously 
recorded. Sensory responses obtained for whisker B2 (A), whisker B4 (B), and fur rostral to B4 (C) are 
illustrated. All PPSTHs were based on 360 single-whisker stimulation trials. Each trial consisted of a 
100-ms step pulse that produced a 3° whisker deflection (upward first). Raw data were smoothed with a 
spline algorithm (Matlab); the statistical significance of sensory responses was computed by means of a 


one-way Kolmogorov-Smirnov test. 
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Summary 


Little is known about the physiological principles that 
govern large-scale neuronal interactions in the mam- 
malian brain. Here, we describe an electrophysiologi- 
cal paradigm capable of simultaneously recording the 
extracellular activity of large populations of single neu- 
rons, distributed across multiple cortical and subcorti- 
cal structures in behaving and anesthetized animals. 
Up to 100 neurons were simultaneously recorded after 
48 microwires were implanted in the brain stem, thala- 
mus, and somatosensory cortex of rats. Overall, 86% 
of the implanted microwires yielded single neurons, 
and an average of 2.3 neurons were discriminated per 
microwire. Our population recordings remained stable 
for weeks, demonstrating that this method can be em- 
ployed to investigate the dynamic and distributed neu- 
ronal ensemble interactions that underlie processes 
such as sensory perception, motor control, and senso- 
rimotor learning in freely behaving animals. 


Introduction 


It is well known that even the simplest of behaviors 
depends on the concurrent activation of large popula- 
tions of neurons, distributed at different levels of the 
neuroaxis. Yet, most of the contemporary neurophysio- 
logical theories still focus on the individual properties 
of single neurons without much consideration given for 
the potential role played by the emergent properties 
of large neuronal ensembles. In part, this is a direct 
consequence of the most common electrophysiological 
approach used to investigate brain function. First intro- 
duced by Adrian (1926), the single electrode recording 
technique aims at sampling the activity of one neuron 
at a time. As a result, the experimental investigation 
focuses on the response properties of single neurons 
and how an individual neuron may encode a given sen- 
sory percept or generate a particular behavior (Barlow, 
1995). Despite its fundamental contribution to modern 
neuroscience, the single neuron recording technique 
severely limits the investigation of the concurrent time- 
dependent interactions between large neuronal popula- 
tions (Vaadia et al., 1995; de Charms and Merzenich, 
1996), since large neuronal ensembles can be studied 
only by obtaining sequential unitary samples (Georgo- 
poulos et al., 1986). Therefore, the lack of adequate 
electrophysiological techniques for neuronal ensemble 
recordings has contributed to the difficulty in character- 
izing the distributed and dynamic interactions between 


large populations of neurons involved in processing and 
storing information (Sherrington, 1906; Hebb, 1949; 
Lashley, 1950; Erickson, 1968; Freeman, 1975; Fuster, 
1995). Here, we describe the implementation of a tech- 
nique for simultaneous, multisite, many single neuron 
recordings and argue that it can be applied to the investi- 
gation of large-scale processing of information by popu- 
lations of a few hundred neurons, located at multiple 
levels of the neuroaxis of behaving animals. 


Results 


A total of 22 rats were used in the experiments described 
here. In these animals, a total of 883 neurons were iso- 
lated by electrode configurations like the ones illustrated 
in Figure 1A. In these configurations, the ideal microwire 
spacing for cortical and subcortical implants was at 
100-250 jm (Figures 1B and 1C); a maximum of 48 
microwires were implanted per animal. This arrange- 
ment allowed simultaneous recordings from neurons in 
the trigeminal ganglion (Vg), principal (PrV) and spinal 
(SpV) nuclei of the trigeminal brain stem complex, the 
ventral posterior medial (VPM) nucleus of the thalamus, 
and the primary somatosensory (SI) cortex (Figure 1D). 
Long-term recordings were obtained by using micro- 
wires with blunt tips (Figure 1B), which proved to be 
much more suitable for long-term chronic recordings 
than electrodes with fine tips. Our multielectrode probes 
proved to be very effective since an average of 2.3 + 0.4 
neurons (mean + SEM) could be isolated per microwire, 
and, for each array of 16 microwires, an average of 
13.9 + 2.1 electrodes (or 86% of the implanted micro- 
wires) yielded at least one discriminable single unit. As 
an example of a typical recording, Figure 2 illustrates 
four distinct extracellular action potentials obtained by 
recording from a single microwire implanted in the rat 
SI cortex. Subsequent analysis of interspike interval, 
poststimulus time histograms, and principal component 
analysis revealed that these waveforms corresponded 
to four independent cortical neurons. Extracellular ac- 
tion potentials in these recordings averaged 73.7 + 
12.1 mV (range: 50 .V-1.5 mV) , while background ,.V 
noise remained at 20 V, indicating that the average 
signal-to-noise ratio of our single-unit recordings was 
around 3.7. 

A Many Neuron Acquisition Processor (MNAP, see 
Experimental Procedures) system (Figure 3) was used 
for all experiments. The configuration of this system was 
fundamental for the performance of population re- 
cordings, since our strategy was aimed at maximizing 
the number of units discriminated in real-time, which 
reduced the time spent on off-line analysis. Analog sig- 
nals (waveforms) could be captured for validation of 
discrimination parameters. With this approach, we could 
perform simultaneous multisite recordings that allowed 
us to reconstruct the parallel flow of sensory information 
across several subcortical and cortical relays of the rat 
trigeminal somatosensory system (Figure 4). 

By using a sequence of 3-D graphs, Figure 4 offers a 


22 


Neuron 
530 


Figure 1. Electrode Configuration for Multilevel Chronic Implants 

(A) Microwire arrays (1) and bundles (2) were designed to maximize the sampling across multiple subregions of the cortical and subcortical 
structures of interest. Arrays (1) were primarily used for cortical and thalamic implants while bundles (2) were used for brain stem implants. 
Field-effect transistors (arrow) were placed in the head stage to correct a mismatch in impedance and to amplify the extracellular signal. 
(B) High power view of a microwire array depicting two rows of eight microwires just before a chronic implant. 

(C) Top view of six microwires forming an array. Notice the Teflon coat (solid arrow) surrounding the stainless steel microwire all the way to 
the blunt tip (open arrow). (D) Example of a multisite implant. Scale bars = (A) 1 mm per division, (B) 250 zm, and (C) 60 ym. 
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way to visualize the spatiotemporal spread of sensory 
information as it ascends from the brain stem to the 
neocortex and how most of the somatosensory system 
is still responding long after the onset (time 0) of a dis- 
crete tactile stimulus. Analysis of multisite recordings 
revealed that, whereas a very restricted sensory re- 
sponse is usually observed in the PrV, the sensory re- 
sponses in the SpV, VPM, and SI cortex cannot be de- 
fined as discrete representations of the cutaneous 
periphery. Figure 5 further supports this finding by dem- 
onstrating that stimulation of different single whiskers 
produced unique spatiotemporal patterns of sensory 
responses from the same ensemble of 30 neurons in 
the SI cortex. Inspection of each of these population 
maps reveals that the same cortical neurons contrib- 
uted, with different response magnitudes and latencies, 
to the responses triggered by the stimulation of different 
whiskers (Figure 5). Only by reconstructing the spatial 
and temporal domains of the ensemble response can 
one reveal the continuous, distributed representations 
that define the location of the tactile stimulus. 

Perhaps the most important result obtained here was 
the demonstration that our ensemble recordings can 
remain stable for long periods. Poststimulus time histo- 
grams (Figure 6A) were used to demonstrate that the 
sensory responses of the same set of cortical neurons 
remained extremely constant for several hours. Stable 
recordings like these demonstrated that our paradigm 
can be used for long-term, real-time monitoring of neu- 
ronal ensemble activity in behaving animals. This is a 
fundamental feature of our paradigm, since it allows one 
to continuously quantitate the modifications induced by 
learning on large populations of cortical and subcortical 
neurons during behavioral tasks. 

Using our paradigm, reliable ensemble recordings 


Figure 2. Real-Time Discrimination of Neu- 
ronal Action Potentials 

Multiple waveforms per microwire were dis- 
criminated in our recording paradigm. Each 
neuron's waveform was identified by using a 
real-time principal component algorithm com- 
bined with a pair of time-voltage boxes. 


were obtained from the same animals for several weeks 
after the implantation surgery. In these experiments, a 
maximum of 100 neurons were obtained when 48 mi- 
crowires were implanted. Since different animals re- 
ceived different numbers of microwires (48, 32, or 16), 
the average number of recorded neurons in our animal 
sample was 41 + 19 (mean + SEM). Further quantitative 
analysis of our results revealed that 9, 13, and 21-30 
days postsurgically, the percentage of microwires that 
yielded recordable single neurons was 86% + 11%, 
90% + 7%, and 85% + 11%, respectively. During the 
same period, the number of neurons isolated per mi- 
crowire also remained very stable or even increased 
slightly: at 9 days postsurgically, 2.14 + 0.47; at 13 days, 
2.39 + 0.32; and 2.92 + 0.26 neurons per microwire 
21-30 days postsurgically. Since most animals were 
sacrificed after a month of recordings, we have not de- 
termined how much longer viable recordings could be 
maintained in rats. However, we observed that in a few 
animals, recordings were maintained for 2 months after 
the initial surgery. Figure 6B provides an example of our 
long-term recordings in rats by demonstrating that the 
same set of microwires provided viable extracellular re- 
cordings over a period of 5 weeks in the same animal. 
Moreover, by using the same approach to record from 
cortical neurons in primates (owl monkeys), we have 
maintained viable recordings of 50-70 single units for 
8-15 months (Nicolelis et al., 1996). Evidently, the possi- 
bility of carrying out such long-term recordings, while 
sampling from the same set of neurons, allows multiple 
experiments to be repeated in the same animal and 
multiple paradigms to be tested in the same experimen- 
tal sample. In addition, it allows us to investigate the 
impact of behavioral training on the properties of neu- 
ronal circuits by continuously studying the physiological 
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Figure 3. Integration of Recording and Behavior Setups 


L_ti 


This schematic diagram describes the organization of our behavior-electrophysiological setup for neuronal ensemble recordings. Notice that 
a single microcomputer controls the 96-channel MNAP and experimental data storage, transmitted from the MNAP to the personal computer 
by the fast-speed MXI bus. The MNAP is responsible for data sampling, waveform discrimination, generation of real-time time signals for 
synchronization of external devices, such as a video camera, and the behavior interface. An output board allows visual and audio monitoring 
of the analog signals. The data files are transmitted through a network for temporary backup storage in an optical drive and subsequently 


for long-term storage in CD-ROMs. A second microcomputer edits behavior protocols and controls the behavior setup. 


attributes of neuronal populations from the onset of 
training until the animals reach performance criterion. 
Finally, it is conceivable that chronic neuronal ensemble 
recordings will permit one to investigate whether neu- 
ronal population activity can be reliably used for control- 
ling prosthetic devices, an area of growing interest in 
the field of biomedical engineering. 


Discussion 


Our results illustrate some of the advantages, as well 
as a few potential applications, of a paradigm for simul- 
taneous, multisite neuronal ensemble recordings in 
behaving animals. By employing this technique, we ob- 
tained very stable recordings of the concurrent extracel- 
lular activity of up to 100 single neurons per animal (48 


microwires implanted, 86% yield, and an average of 
2.3 neurons/microwire), dispersed along the entire rat 
neuroaxis, with a time resolution of about 250 js anda 
total tissue sampling area per site of about 2 mm*. The 
combination of these features, particularly the high tem- 
poral resolution, the longevity of the recordings, and 
their distributed nature cannot be matched by any other 
electrophysiological (e.g., EEG, field potentials, or sin- 
gle-unit recording) or imaging technique (optical re- 
cording of intrinsic signals or voltage-sensitive dyes) 
designed for the study of large-scale brain activity in 
experimental animals. 

Another contribution of the technological approach 
described here is the considerable reduction in com- 
puter requirements for carrying out large-scale en- 
semble recordings. While alternative approaches may 


Figure 4. Multilevel Recordings in Behaving Animals 


Simultaneous multilevel neuronal ensemble recordings allowed us to quantify the spatiotemporal spread of neuronal activation across many 
processing levels of the rat trigeminal somatosensory system, following stimulation of a single whisker. At each level of the pathway, 3-D 
graphs were used to represent patterns of neuronal ensemble activity. In each of these graphs, the x axis represents the poststimulus time 
(in ms), the y axis represents the number of neurons recorded at each level, and the color-coded gradient in the z axis was used to represent 
the response magnitude of the neurons (in spikes/s). In this scheme, dark red indicates the highest firing (140-150 spikes/s), and dark green 
indicates the lowest firing (0-10 spikes/s). SI, somatosensory cortex; VPM, ventral posterior medial nucleus of the thalamus; SpV, spinal 
nucleus of the trigeminal brain stem complex; and PrV, principal nucleus of the brain stem trigeminal complex. 
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Figure 5. Distributed Responses in the Rat Somatosensory Cortex 


In each of the 3-D graphs depicted here, simultaneous neuronal ensemble recordings were employed to quantify the spatiotemporal spread 
of cortical responses following the stimulation of a single whisker (identified on the top of the graph). The x axis represents poststimulus time, 
and the z axis represents response magnitude in spikes/s. All neurons were located in the infragranular layers of the primary somatosensory 
cortex. 


require a large number of microcomputers to be syn- As required for any paradigm designed for the study 
chronized (Wilson and McNaugton, 1994), in our para- of neuronal ensembles, our recordings allow large sam- 
digm, a single personal computer is used to control ples of neurons to be monitored per animal. The use of 
the entire electrophysiological setup and store neuronal microwire arrays was fundamental to achieve this goal. 
spiking data, while a second microcomputer, interfaced While neuronal ensemble recordings have been ob- 


to the MNAP, controls the behavior setup. tained acutely in the past (Gerstein and Clark, 1964; 
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(A) Poststimulus time histograms depict the sensory responses of a set of six cortical neurons at two distinct times. 


(B) Microwires provided viable recordings for several weeks. 


reviewed by Kruger, 1983), most studies aimed at ob- 
taining chronic recordings of single units have employed 
microwire bundles (Kubie, 1984; Shin and Chapin, 1990). 
The main reason for this choice is the stability and reli- 
able yield of single neurons that can be obtained with 
microwires. Here, data obtained with microwire arrays 
further demonstrate that in regions of moderate cell 
density, such as the cortex, thalamus, and trigeminal 
brain stem complex, one does not need to employ more 
complex electrode configurations, such as tetrodes 
(Wilson and McNaugton, 1994; Gray et al., 1995), for 
successfully carrying out neuronal ensemble record- 
ings. The difficulty in manufacturing tetrodes, and the 
fact that no group has so far provided quantitative data 
to demonstrate that these electrodes maintain high neu- 
ronal yield (15-20 neurons per tetrode) for many weeks, 
further justify the use of arrays of isolated microwires 
as the most efficient solution for carrying out chronic 
neuronal ensemble recordings. New multichannel elec- 
trode designs have recently appeared in the literature 
(Hoogerwerf and Wise, 1994; Nordhausen et al., 1994), 
but there is still scant evidence to support their use in 


long-term chronic recordings. In fact, we and others 
have recently demonstrated that by using microwire 
arrays, reliable long-term neuronal ensemble recordings 
can be obtained in many mammalian species currently 
used in neurophysiology. Those include adult and young 
ferrets (Nicolelis et al., unpublished data), owl monkeys 
(Nicolelis et al., 1996), and rhesus monkeys (Perepelkin 
and Schwartz, 1996). 

Many neuroscientists (Sherrington, 1906; Hebb, 1949; 
Lashley, 1950; Lilly, 1958; Erickson, 1968; Somjen, 1972; 
Freeman, 1975) foresaw that to understand how sensory 
information is processed, converted into percepts, 
stored into memory, and then used to generate behav- 
iors, one needed to comprehend the physiological prin- 
ciples that underlie large-scale interactions among 
widely distributed and interconnected populations of 
neurons. The available experimental evidence has sup- 
ported this view by indicating that most fundamental 
brain functions involve highly dynamic and distributed 
neuronal interactions (Freeman, 1983; Georgopoulos et 
al., 1986, 1993; Nicolelis and Chapin, 1994; Wilson and 
McNaugton, 1994; Fuster, 1995; Nicolelis et al., 1995; 
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Welsh et al., 1995; Deadwyler et al., 1996). Therefore, 
the results described here further support the notion 
that, as neuronal ensemble recordings are incorporated 
into the arsenal of current neurophysiological tech- 
niques and explored to their full potential, we will have, 
at last, a realistic chance of reconstructing the neuronal 
engram, a task which, as put by Somjen (1972), has 
eluded us all. 


Experimental Procedures 


Microelectrode Design 

Different configurations of microelectrode matrices were specially 
designed (NB Laboratories, Dennison, TX) to allow simultaneous 
recordings of the extracellular activity of large numbers of single 
neurons, distributed across up to five distinct neuronal structures 
that define the rat somatosensory system. In all experiments de- 
scribed here, stainless steel, Teflon-coated microwires (50 j.m in 
diameter, California Fine wire) were used to build different micro- 
electrode matrices, each of which was designed to maximize the 
number of neurons sampled throughout the spatial domain of differ- 
ent cortical and subcortical structures. For instance, microelectrode 
arrays, containing two parallel rows of eight microwires each, were 
used for chronic implants in the rat primary (SI) somatosensory 
cortex. In these arrays, the distance between the microwire rows 
varied from 0.5-1 mm, and the distance between pairs of microwires 
in a row varied from 100-250 j1m (Figure 1). Another commonly used 
configuration was the microelectrode bundle, containing either 8 or 
16 microwires staggered at different lengths, which were primarily 
used to record from neurons in deep structures, such as the ventral 
posterior medial nucleus of the thalamus and the different subnuclei 
of the trigeminal brain stem complex. Arrays and bundles were built 
by first soldering the microwires to a 20-pin plastic connector, then 
covering the connections with a thin layer of epoxy. Two ground 
wires were also soldered to free pins in the plastic connector. Finally, 
polyethylene glycol was used to mold the arrays and bundles and 
provide them with enough stiffness to be implanted in the brain. 


Surgical Procedure 

Microwire arrays and bundles were implanted during stereotaxic 
surgeries. For these procedures, rats were first anesthetized with a 
single injection of pentobarbital sodium (50 mg/kg, |.P.) and then 
transferred to a stereotaxic apparatus. Supplementary doses (one 
third of the original) of pentobarbital were given when necessary. 
The head was shaved and then cleaned with a betadine solution. 
A single midsagittal incision of the skin was used to access the 
surface of the skull. Following retraction of soft tissue, the perios- 
teum of the dorsal surface of the skull was completely removed, 
and the bone surface was scrubbed with saline until no sign of 
blood was left. The surface of the skull was then dried, and a series 
of small craniotomies were produced with a high speed dental drill 
at the stereotaxic coordinates required for the implantation of elec- 
trode arrays in the Vg, PrV, SpV, VPM, and SI cortex. A second 
series of four to six craniotomies were open so that stainless steel 
screws could be firmly attached to the skull. These screws were 
used for securing probes and for grounding purposes. Just before 
the implantation, microwires forming bundles were cut to the ideal 
length, using a sharp pair of scissors, and then soaked in a supersat- 
urated solution of sucrose. Special care was taken not to allow the 
sucrose to cover the microwire tips. After drying, the arrays and 
bundles were mounted in the holder of a hydraulic micropositioner 
(Kopf, Tujunga, CA) and subsequently slowly driven (~100 m/min) 
into the brain. Although the 16-microwire arrays could be driven 
through the dura most of the time, in some instances, a small slit 
was made in the dura overlying the SI cortex to facilitate the implant. 
Single and multiunit activity were monitored continuously through- 
out the surgery to help locate the position of each electrode. During 
these recordings, the receptive field of each unit was qualitatively 
characterized until the arrays were positioned in the structure of 
interest. After reaching the final target, the ground wires of each 
connector were wrapped around one or two of the metal screws 


previously implanted in the skull. The craniotomy was then sealed 
with a layer of agar (4% in saline). Once the agar solidified, dental 
cement was applied until it reached the top border of the plastic 
connector. This procedure was repeated until all arrays and bundles 
had been implanted. The end result of one of these implants is 
illustrated in Figure 1D. To finish the procedure, the skin was loosely 
sealed around the probes and the animal was transferred to a recov- 
ery cage. The analgesic Buprenorphine (0.1-0.5 mg/kg S.C. every 
8 hr) was administered for 2-3 days while the animal recovered from 
the surgery. Electrophysiological recordings started 5-7 days after 
the surgery. Experiments were carried out in a recording chamber 
using anesthetized and freely behaving animals. 


Instrumentation for Real-Time Ensemble Recordings 

Simultaneous recordings of large samples of single neurons were 
carried out by an MNAP (Spectrum Scientific, Dallas; see Figure 3). 
In the most complete configuration, the MNAP allows simultaneous 
sampling from 96 microwires and discrimination of up to four individ- 
ual action potential waveforms per wire, for a maximum of 384 
recorded neurons. In this configuration, head stages (NBLABS, Den- 
nison, TX), containing 16 field-effect transistors (FETs, Motorola 
MMBF5459) arranged in two rows of eight at the end of TVC-insu- 
lated cables, are used to connect the 20-channel plastic connectors 
cemented in the animal's head to the MNAP preamplifiers (Figure 
1D). In all recordings, the FETs were set as voltage followers with 
unit gain. The MNAP preamplifiers contain differential OP-Amps 
(gain 100; bandpass 100 Hz-16 kHz). Their output signals are trans- 
mitted, through ribbon cables, to 96 A/C-coupled differential amplifi- 
ers on 6 input boards, each containing 16 amplifiers. Once in the 
input boards, the analog signals pass through the first level of ampli- 
fication (jumperable gain of 1, 10, or 20), are filtered (bandpass 400 
Hz-8 kHz), and reach the final stage of amplification (programmable 
multiplier stage, ranging from 1-30). These boards also include one 
12-bit analog-to-digital (A/D) converter per channel, which simulta- 
neously digitizes the waveforms defining extracellular action poten- 
tials at 40 kHz. After A/D conversion, the signals are routed to DSP 
boards, each of which contain four digital signal processors (DSP, 
Motorola 5602) running at 40 MHz (instruction read at 20 MHz). Each 
DSP handles data from eight input channels and contains 32 K 24 
bits of SRAM and 4 K 16-bit words of dual port SRAM memory. A 
timing board is responsible for distributing timing and synchroniza- 
tion signals to the entire MNAP. It also provides a digital time output 
that is used to synchronize external devices or to drive a video timer 
for a professional VCR used to store video records of the animal’s 
behavior. The DSP boards also provide inputs for sampling digital 
pulses generated from behavioral cages. A single host Pentium 
microcomputer (100-200 MHz, with 64 MB of RAM and 2.1 GB of 
disk space) running a C++ software (SSCP, Spectrum Scientific, 
Dallas) in the Windows 3.11 operating system (Microsoft, Seattle) 
controls the MNAP over a serial line. Spike discrimination programs 
are downloaded from the PC host to the DSPs. Single spikes are 
discriminated by combining a modified version (Nicolelis and 
Chapin, 1994) of a principal component algorithm (Abeles and 
Goldstein, 1977), running in real-time, and one pair of time-voltage 
windows per unit. The size and position of the time-voltage boxes 
are defined by the experimenter to isolate the waveforms that be- 
longed to a given unit. Spikes are only accepted as valid when 
they pass through both boxes. During the experiment, the time of 
occurrence of each of the valid spikes for all 96 channels is trans- 
ferred to the hard disk of the PC host through a parallel bus (MXI- 
Bus, National Instruments, CA), which can transfer 2 MB of data/s. 
Digitized samples of the spike waveforms are also recorded periodi- 
cally and stored for off-line analysis using a visualization program 
(SpikeWorks, Spectrum Scientific, Dallas). Optical drives (1.2-2.4 
GB cartridges, Pinnacle Inc., CA) are used for temporary backup of 
the data files. These files are then transferred through an ethernet- 
based network to a computer server containing a CD-Recorder 
(PinnacleMicro, CA), which produces permanent records of the data 
in CD-ROMs. The MNAP also supplies options for analog backup 
using 8-16 channel tape recorders. In all experiments, low threshold 
tactile stimulation (step pulses of 100 ms in duration, delivered at 
1-10 Hz) is provided by a computer-monitored Grass $8800 stimula- 
tor, which drives a vibromechanical probe. The digital outputs of 
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the stimulator were also transmitted to the MNAP to be stored with 
the spike data. 


Population Data Analysis 

Neural ensemble data were processed off-line using a series of 
graphics, statistical, and numerical analysis packages. The first level 
of analysis involved the production of single cell poststimulus time 
and cumulative histograms, which were used for defining the spatio- 
temporal receptive fields of single neurons. All of these preliminary 
steps of data analysis were carried out by using a spike train analysis 
software (Stranger, Biographics, Winston-Salem, NC) specially de- 
signed to handle large neuronal ensemble data sets. Spatiotemporal 
reconstructions of single neuronal receptive fields and’ population 
maps were carried out by using a graphics interface software, devel- 
oped in C++, combined to MATLAB (Mathworks, MA). Finally, sta- 
tistical packages like CSS-Statistica (Statsoft, Tulsa, OK) and SPSS 
(SPSS Inc., Chicago) were used for applying multivariate statistical 
methods to the analysis of neuronal ensemble data. 
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We used simultaneous multi-site neural ensemble recordings to investigate the representation of 
tactile information in three areas of the primate somatosensory cortex (areas 3b, SII and 2). Small 
neural ensembles (30-40 neurons) of broadly tuned somatosensory neurons were able to identify 
correctly the location of a single tactile stimulus on a single trial, almost simultaneously. 
Furthermore, each of these cortical areas could use different combinations of encoding strategies, 
such as mean firing rate (areas 3b and 2) or temporal patterns of ensemble firing (area SII), to repre- 
sent the location of a tactile stimulus. Based on these results, we propose that ensembles of broadly 
tuned neurons, located in three distinct areas of the primate somatosensory cortex, obtain informa- 
tion about the location of a tactile stimulus almost concurrently. 


One of the most basic functions of the nervous system is the 
localization of sensory stimuli. Information about stimulus loca- 
tion is, of course, available in the peripheral receptor array, whose 
activity may be processed along divergent paths and levels of 
central pathways. For instance, the translation of this receptor 
activity through the thalamocortical circuitry may produce a 
sensation of localization, whereas translation of the same activ- 
ity through the spinal circuitry may produce a reflex movement 
toward or away from the stimulus location. How might the brain 
actually combine information available in many neural struc- 
tures to produce a perceptual indication of the location of a tac- 
tile stimulus? In the complex somatosensory system of primates, 
which includes many reciprocally connected cortical areas!, a 
fundamental question is whether information about tactile stim- 
ulus localization is preserved throughout these many cortical 
areas, or whether such information is conserved within partic- 
ular cortical populations that then serve as a general reference. 
This second alternative seems to be suggested by the degrada- 
tion of topographic maps of body surface as one proceeds from 
early to late stages of cortical processing!. Clearly, information 
about stimulus location is well preserved in the precise topo- 
graphic maps in the brainstem, thalamus and primary 
somatosensory cortex. Yet, is this information necessarily lost 
in other cortical areas where neurons have large receptive fields? 
An alternative possibility is that information about stimulus 
location could be transformed from a spatial code, easily recog- 
nized at a single-neuron level, to a distributed code, best recog- 
nized in the simultaneous activities of large populations of 
cortical neurons. It is also conceivable that at late stages of cor- 
tical processing, the location of a stimulus could be represent- 
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ed in the temporal patterns of neural ensemble firing. In fact, 
research on artificial neural networks demonstrated that a ‘tem- 
poral encoding’ strategy emerges as a signature of recurrent dis- 
tributed systems that can resolve noisy patterns. This temporal 
coding could manifest itself through precise sequences of action 
potentials, synchronous neuronal firing or time-dependent mod- 
ulation of the neural ensemble firing rate. 

Such issues are fundamental to our understanding of how the 
primate cerebral cortex operates and are not restricted to the spe- 
cial functions of the somatosensory system. In the auditory cor- 
tex, for instance, the lack of an obvious spatial map for sound 
location has led some to propose that populations of auditory 
cortical neurons may encode location in a complex distributed 
fashion that does not depend on precise spatial mapping at the 
single-neuron level°. To date, however, little attention has been 
paid to the possibility that such nonspatial codes could effectively 
represent tactile stimulus location in the somatosensory system. 
Indeed, the secondary somatosensory cortex and fields of the 
posterior parietal cortex have long been considered to be unin- 
volved in stimulus location coding, because the large receptive 
fields of neurons in these regions! would seem to preclude the 
accurate representation of this information. This view could be 
challenged by evidence that accurate stimulus location informa- 
tion can be extracted from the collective responses of neural 
ensembles in these cortical areas, as proposed by distributed 
models of cortical function®!?. 

To address this possibility, in this study we used large-scale, 
chronic, multi-site neural ensemble electrophysiology!*»'!* to 
simultaneously record the sensory responses of up to 135 single 
neurons located in three cortical somatosensory areas (areas 3b, 
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Fig. 1. Location of chronically implanted microwire arrays used to 
record neuronal ensemble activity in the somatosensory and motor 
cortex. In this schematic representation of the somatosensory and 
motor areas of the neocortex of the owl monkey, each colored rec- 
tangle depicts the anatomical location of a chronically implanted array 
containing 16 microwires (organized in two rows of eight). Note that 
Sl is buried within the lateral sulcus. 


SII and 2) in adult owl monkeys (Aotus trivirgatus)!>-'7. This 
species was chosen because the cortical areas of interest are 
accessible on the dorsal surface of a relatively smooth neocortex, 
and much is known about the organization and connections of 
its cortex!>. 


Results 

In each of the three monkeys used in this study, about 80% of 
the microwires yielded at least one discriminable single unit, and 
up to four neurons per electrode could be discriminated in each 
animal. Chronically implanted microwire arrays provided stable 
neural population recordings for 8 months in monkey 1, 19 
months in monkey 2 and 5 months in monkey 3. During these 
recording periods, a maximum of 135 (monkey 1, 48 microwires), 
104 (monkey 2, 48 microwires) and 77 (monkey 3, 32 
microwires) cortical neurons were simultaneously discriminat- 
ed in a single recording session. Different samples of neurons 
were obtained in each recording session, although a given sample 
of neurons was often kept for many days. Histological analysis 
showed that the neurons recorded in this study were located pri- 
marily in the infragranular layers of the following cortical areas 
(Fig. 1): for monkey 1, primary motor cortex (MI), secondary 
somatosensory cortex (SII) and posterior parietal area 2; for mon- 
key 2, supplementary motor area (SMA), area 3b, SII and area 2; 
and for monkey 3, MI, SII and area 2. Only results from the 
somatosensory areas 3b, SII and 2 are discussed here. 

Broadly tuned neuronal responses to tactile stimulation were 
observed in the infragranular layers of the primate somatosen- 
sory cortex (Fig. 2). Raster plots and corresponding post-stim- 
ulus time histograms (PSTHs) were used to depict the sensory 
responses of a sample (6 out of 30) of simultaneously-recorded 
SII neurons following the stimulation of different hand locations. 
As a rule, individual SII as well as area 2 neurons responded to 
the stimulation of multiple digits and regions of the dorsal surface 
of the hand (Fig. 3). Quantitative analysis of PSTHs in both awake 
and lightly ketamine-anesthetized (10-15 mg/ kg) animals 
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demonstrated that virtually all neurons in the infragranular lay- 
ers of areas SII and 2 showed large multi-digit receptive fields 
(RFs). These RFs were considerably larger than those observed 
in layer IV of area 3b, and similar to the large neuronal RFs pre- 
viously reported in rhesus monkeys'®!7, The raster plots (Fig. 2) 
also illustrate the difficulty of identifying which skin site was 
stimulated when only a single trial of single neuron’s response 
was considered (see neurons 22a, 25 or 30a, for instance). In fact, 
for most neurons, the location of the stimulus could not be iden- 
tified by examining either single-trial sensory responses or aver- 
aged information from many trials depicted in PSTHs (Fig. 2). 
In addition, the areal location of a cortical neuron could not be 
established by simple inspection of individual neuronal PSTHs 
(Fig. 3). This is because most somatosensory cortical neurons 
showed similar averaged sensory responses that could not be dis- 
tinguished in terms of minimal latency, duration or frequency of 
responses. This similarity in intrinsic temporal response charac- 
teristics of somatosensory neurons is illustrated by the compar- 
ison of a sample of simultaneously recorded SII and area 2 
neurons, which showed virtually identical PSTHs in response to 
punctate tactile stimulation (Fig. 3). 

In general, the multidigit RFs of SII and area 2 cortical neu- 
rons were characterized by the presence of a center whose stim- 
ulation elicited the largest sensory response and the shortest 
response latency. Stimulation of surround regions of the RF 
induced smaller but statistically significant excitatory respons- 
es at longer response latencies, as previously reported in rats!8, 
In addition to having multi-digit RFs, area 2 neurons could be 
driven by stimulation of the face and the back of the hand 
(Fig. 4c and d). By combining the sensory responses of indi- 
vidual neurons into population maps, we were able to recon- 
struct the neural ensemble representations of punctate tactile 
stimuli across the somatosensory cortical areas (Fig. 4) in the 
same animals. Punctate stimulation of different parts of the 
hand and/or face produced distinct spatiotemporal patterns of 
neuronal activation in SII (Fig. 4a and b) and area 2 (Fig. 4c 
and d). These distinct profiles were generated by variations in 
the response magnitude and latency of individual neuronal sen- 
sory responses. Note, however, that many neurons contributed 
to the population response resulting from a given tactile stim- 
ulus, suggesting that distributed representations exist in areas 
SI and 2 of the primate somatosensory cortex. 

Further analysis of the population maps for all three cortical 
areas recorded in each animal revealed a considerable latency 
overlap between the sensory responses of neurons (measured 
with 1-ms accuracy). In an example recording (Fig. 5a), this activ- 
ity overlap was observed during stimulation of the tip of the D2 
digit in monkey 3. This stimulus initially triggered a sensory 
response in neurons located in SII (10-15 ms), and a couple of 
milliseconds later massive neural firing was observed in both SI] 
and area 2. Comparison of minimal response latencies for all 
neurons recorded in areas 3b, SII and 2 (Fig. 5b) revealed a sig- 
nificant degree of overlap. 


ENSEMBLE CODING OF TACTILE STIMULUS LOCATION 

Our overall strategy in this analysis was to measure how well the 
firing patterns of cortical ensembles could predict, on a single 
trial, the location of a punctate tactile stimulus applied to the 
animal’s body. Because many skin sites were stimulated in ran- 
dom order, multiple combinations involving 3, 5 or 10 sites were 
used for this analysis. Ten-fold cross-validation was employed 
for each analysis to yield the highest stability and accuracy of the 
results. For each combination of sites, 10% to 90% of the original 
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Fig. 2. Single-neuron responses to tactile stimulation at different sites on the hand. Raster plots and post-stimulus time histograms were 
used to illustrate the sensory responses of a subset of simultaneously recorded SII neurons following the punctate tactile stimulation of seven 
neighboring regions of the monkey hand. All single SII neurons (identified on top of each column) tended to respond significantly to all the 
stimulated skin sites. In addition, each neuron’s averaged responses (see histograms) were similar across these skin sites. Inspection of single 
trial responses for a given single neuron did not allow identification of the site of stimulation, either because the response was indistinguish- 
able to that obtained for other sites or because the neuron did not fire in that trial. 


nature neuroscience * volume 1 no7 * november 1998 623 


33 


articles 
Sil Area 2 
dsp17 4 dsp29b : dspOla 6 dsp05d 
3 
2 2 8; 4 
1 44 2 
o3_ 0. ie oa lid of uu boris hy 
dsp18¢ dsp30a dsp01b dsp10a 
1s @ 123 6 P 
10 2 8 43 
5 20 4 23 
ition ss 0: 0 Ostin i im tt 
3 dsp19 dsp30b dsp03a dsp10c 
2 4 8 6. 
3 1 2 £ 4 
2 See ee 0 a! Fa 0. i od led! ds i 
a dsp24c 6.  dsp30d - dsp03b 20. dspi2c 
- 5 8 15 
o a 6 
- ey 
2 oo 01! 2 Bild i 8) sa sansa. 
el  aenananen manenanen | 
s) 
3 dsp2s 6. dsp3tc zg dsp04a 4, osp13a 
wy 8 4 3 
= 4 2 ¥ 4 - | 
& 08s tee llly 03 etl 1 3 ad avoh oft | | Mat 
dsp26a dsp31d dsp04b dsp15b 
6 s 12 
4 } 4 8 
2 | 2 4 
0. mite ere Oss Nhe wns ' 3 aad, ba lll! 
Vrs  anenenen meneame! ' anananen nenanenan aan maanande! 
dsp26b dsp32 dsp0Sa dsp16 
. 3 | 15 
2 2 Fs 10. 
0 i \ iL 63 ved vuole i oe 
-0.05 0 -0.05 0 0.05 -0.05 i) 0.05 -0.05 0 0.05 
Time (s) Time (s) Time (s) Time (s) 


Dorsal hand stimulation 


Fig. 3. Post-stimulus time his- 
tograms depict the simultaneously 
recorded sensory responses of 
ensembles of neurons in areas SII and 
2 following the punctate tactile stim- 
ulation of hair located in the dorsal 
hand of a monkey. Notice that all 
neurons in these two areas 
responded to this stimulus (as well as 
stimulation of many other sites, not 
shown). Inspection of these and 
other histograms did not reveal any 
clear encoding strategy. For instance, 
no difference in the frequency of 
response can be observed between 
SIl and area 2 neurons. Moreover, 
both populations showed overlap- 
ping minimal response latencies. 
Finally, the duration of the single- 
neuron sensory responses in both 
areas was similar. Therefore, no dif- 
ference in intrinsic temporal 
response characteristics between 
these two populations of neurons 
was observed in these post-stimulus 
time histograms. 
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360 stimulus trials obtained per skin site were used to ‘train’ dif- 
ferent artificial neural networks or to derive discriminant func- 
tions using linear discriminant analysis. Once the training phase 
was completed, the trained artificial networks (or the discrimi- 
nant functions, in the case of linear discriminant analysis) were 
used to identify which of the skin sites was stimulated in each of 
the remaining ‘testing’ trials. Testing trials were never presented 


Fig. 4. Distributed responses 
of area SII and 2 neural ensem- 
bles. Population maps depict 
the simultaneously recorded 
sensory responses of neural 
ensembles located in areas SIl 
and 2 of the owl monkey 
somatosensory cortex. (a, b) 
Stimulation of the proximal 
phalanx of digit | and of the 
dorsal surface of the hand elicit 
distinct spatiotemporal pat- 
terns of response in the SIl 
cortex. (c, d) Stimulation of 
the face and dorsal surface of 
the hand elicit distinct spa- 
tiotemporal patterns _— of 
ensemble activity in area 2. In 
each of the three-dimensional 
plots, the x-axis depicts the 
simultaneously recorded neu- 
rons, the y-axis is post-stimulus 
time in milliseconds, and the z- 
axis is the response magnitude 
in spikes per second. 
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during the training phase of the analysis. Both testing and train- 
ing trials were presented in a random sequence, and equal num- 
bers of trials per site were used. 

As a way to cross-validate our results, different analysis para- 
meters were varied. First, three methods for statistical pattern 
recognition (linear discriminant analysis, learning vector quan- 
tization and back-propagation) were applied to the same data 
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Fig. 5. Concurrent activation of somatosensory cortical areas following punctate tactile stimulation. (a) Color-coded three-dimensional 
matrices depict the post-stimulus firing of simultaneously recorded neurons in the SII cortex (top panel) and area 2 (bottom panel) following 
the stimulation of a finger tip. These plots depict the response of each neuron according to its position in the 2 X 8 electrode arrays 
implanted in the two cortical areas. Data were plotted at 5 millisecond post-stimulus time intervals. At each time epoch, the two rectangu- 
lar panels, separated by a vertical white line, were used to represent the spatial extent of neural activity. The y-axis of each rectangular panel 
represents the rostrocaudal position of the microwires (top rostral-most electrodes | and 9; caudal-most wires 8 and 16), the left rectangle 
depicts the lateral component of the array, and the right panel depicts the medial component of the array. Because up to four neurons were 
recorded per microwire, the x-axis of each rectangle contains up to four slots per microwire to represent the response of each of the 
recorded neurons. The z-axis represents variations in neuronal response magnitude (dark red represents >4 standard deviations above spon- 
taneous firing, and blue represents spontaneous firing, in spikes per second). Notice that following a small activation of the SII cortex at the 
10—15-ms epoch, there is concurrent activation of both cortical areas (SII and 2). (b) Minimal latency distributions for single neurons in each 
of the three cortical areas. The minimal latencies calculated for data obtained in two monkeys demonstrate that following an initial activation 
of both area 3b (minimal response latency, 9.0 + 2.2 ms) and SII (12.5 + 3.52 ms), concurrent activation was observed in these areas and area 
2 (minimal response latency, |7.8 + 5.0 ms). 
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sets. Second, three unique formats of data input were used for _ location in 72.5 + 3.5% of the individual testing trials (Fig. 7a 
each pattern recognition analysis: raw spike trains, principal com- _—_ and b). The same analysis revealed that SII ensembles provided 
ponents!” and independent components” derived from the raw _ the correct identification of the stimulus location in 94.5 + 0.6% 
spike trains. Finally, different training-set sizes (90%, 50% and _ of the single testing trials in monkey 1 (40 neurons, not shown), 
25% of the total number of trials) were used in the analysis. 70.0 + 1.9% of the trials in monkey 2 (26 neurons, Fig. 7a and b) 
Through all of these variations in analytical approach, statisti- | and 73.3 + 1.6% in monkey 3 (30 neurons, Fig. 7c and d). 
cally similar results were obtained (Fig. 6a and b), indicating that | Above-chance discrimination was also obtained when area 2 
neither the pattern recognition analysis method (Fig. 6a, p< 0.77, ensembles (48.0 + 1.8%, 34 neurons, monkey 3, Fig. 7c and d) 
MANOVA), nor the type of data preprocessing, nor the training —_ were tested independently. 

schedule influenced the final outcome of the population analy- The potential encoding mechanisms used by each cortical 
sis. The relative proportion of training to testing trials was also —aarea were investigated by, first, sequentially removing neurons 
generally unimportant for these results, although significant (‘neuron dropping’) from each cortical ensemble to measure the 


reductions in discrimination performance by the artificial neur-_ variation in discrimination capability as a function of the ensem- 
al networks were obtained when the number of training trials _ ble size; second, independently shuffling the trial order for each 
was reduced to 10% of the total (p < 0.03, Fig. 6b). neuron, a maneuver aimed at testing the robustness of intertrial 

An artificial neural network based on learning vector quan- _ neuronal correlations; and third, varying the integration time 


tization was used to evaluate the discrimination capabilities of | used to describe each neuron’s sensory response (from 3 to 45 
simultaneously recorded cortical neural ensembles located in _ms), a procedure that altered the temporal structure of each neu- 
areas 3b and SII (monkey 2, Fig. 7a and b) and areas SII and2 _ron’s sensory response (‘bin clumping’). 


(monkey 3, Fig. 7c and d). A total of 4 (monkey 1), 10 (monkey The neuron-dropping procedure further supported the exis- 
2) and 25 (monkey 3) unique combinations of 3 skin sites were _ tence of distributed representations of tactile information in each 
used to quantify each cortical ensemble’s discrimination capa- _ of the three cortical areas, by demonstrating that removal of indi- 
bility. This analysis revealed that, when independently tested, _ vidual ‘best predictor’ neurons from the ensemble produced only 


statistically significant single-trial discrimination of the location small and gradual reductions in the correct classification of sin- 
of a tactile was achieved using cortical ensemble firing patterns _ gle trials by the neural ensemble, as shown for four neural ensem- 
from each of the three areas investigated (3b, SII and area2) and _ bles (Fig. 8a and b). SII functions (from two monkeys) were very 
a variety of skin site combinations. For example, the firing pat- _—_ similar to each other but differed when compared to those 
tern of an ensemble of 24 neurons located in area 3b of monkey _ obtained for areas 3b and 2 ensembles. Even though area 3b 
2 uniquely specified the correct identification of the stimulus ensembles outperformed area 2 populations of the same sizes, 
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Fig. 6. Multiple statistical pattern recognition 
approaches cross-validate results of neural ensemble 
performances. (a) Use of three pattern recognition 
algorithms (learning vector quantization, non-linear 
back-propagation network and linear discrimination 
analysis) and three types of data input format (raw spike 
data, principal component, PCA, and independent com- 
ponents, ICA) produced similar results (pb < 0.77, 
MANOVA). (b) Similarly, no statistical difference was 
observed when 25-90% of the trials were used to train 
the artificial networks (learning vector quantization, 
LVQ, and back propagation, NLB) and derive the dis- 
crimination functions (linear discriminant analysis, 
NLB LDA). Because statistically significant reductions in dis- 
crimination could occur when only 10% of the trials 
were used, this training routine was avoided. 


eye 


the two cortical areas showed very similar gradual decays in sin- 
gle-trial discrimination performance as neurons were removed 
from the ensembles. Increasing the number of skin sites used in 
the analysis (from 3 to 10 sites) reduced discrimination perfor- 
mance. However, above-chance discrimination was observed for 
each combination of sites. Overall, most individual neurons alone 
were poor discriminators of the stimulus location on a single 
trial, as the vast majority of the single 3b (92%), SII (96%) or 
area 2 (100%) neurons tested could not be used to classify more 
than 50% of the trials correctly for combinations of 3 skin sites. 

Independent shuffling of trial order for each single neuron 
had little effect on the overall classification of single trials by either 
artificial neural networks or linear discrimination analysis. This 
suggested that the neuronal responses were tightly locked to the 
presentation of the tactile stimuli throughout the duration of 
each recording session. In some cases, the performance of arti- 
ficial neural networks improved slightly for certain skin site com- 
binations after trial shuffling, which could suggest the existence 
of correlated ‘biological noise’ embedded in the single-neuron 
responses on each trial. Furthermore, this finding suggests that 
effective single trial discrimination of punctate tactile stimulus 
location does not require precise (1-2 ms accuracy) intra-trial 
neural ensemble spiking sequences!”, extensive neuronal syn- 
chronization”’ or covariance across the population of simulta- 
neously recorded neurons. 


DIFFERENCES IN ENCODING STRATEGY BETWEEN CORTICAL AREAS 
Both temporal- and rate-based codes”* have been proposed to 
account for the ability of cortical neural ensembles to represent 
sensory information. In the final step of our analysis, we inves- 
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tigated which of these encoding schemes could best explain the 
single-trial representation of tactile stimuli in the somatosenso- 
ry areas 3b, SII and 2. First, we studied whether the temporal 
structures of the sensory responses of neurons in each cortical 
area were involved in the encoding of stimulus location. To 
address this issue, we examined the artificial neural network's 
ability to classify correctly the location of a stimulus on a single 
trial using bin clumping, by integrating neuronal firing data into 
a range of bin sizes (from 15 3-ms time bins to a single 45-ms 
bin) used to represent the first 45 ms of post-stimulus period, as 
shown for pairs of cortical areas recorded in two monkeys (2 and 
3; Fig. 9). Bin clumping significantly reduced the discrimination 
capability of SII neural ensembles in all three monkeys: from 94.5 
+ 0.6% to 90.3 + 1.1% in monkey 1 (not shown), from 70.0 + 
1.9% to 61.2 + 2.1% in monkey 2 (p < 0.001; Fig. 9a) and from 
73.3 + 1.6% to 59.5 + 1.7% in monkey 3 (p < 0.0001; Fig. 9b). 
Bin clumping, however, had no effect on the discrimination capa- 
bility of the 3b ensemble in monkey 2 (72.5 + 3.5% using 3-ms 
bins and 72.0 + 3.7% using a single 45-ms bin, p < 0.8; Fig. 9c) or 
the area 2 ensemble recorded in monkey 3 (48.0 + 1.8% using 3- 
ms bins and 48.7 + 1.5% using a single 45-ms bin, p < 0.72; Fig. 
9d). Even though single SII neurons have been reported to 
respond well to time-varying stimuli, particularly high-frequen- 
cy vibrotactile stimuli”, no basic differences in intrinsic tempo- 
ral response characteristics were observed in the present study 
between SII neurons and neurons recorded simultaneously in 
either areas 3b or 2. (See Fig. 3 for a comparison between areas SII 
and 2.) A plausible interpretation of these bin-clumping results is 
that the reduction in single trial discrimination by SII ensembles 
resulted from the disruption of time-dependent patterns of neur- 
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Fig. 7. Single-trial discrimination by 


a ein b neural ensembles located in three 
90 somatosensory cortical areas (3b, SII and 

90 area 2) as measured by an artificial neural 

60 network trained with a learning vector 


quantization algorithm. (a) Neural ensem- 
bles in areas 3b and SIl (monkey 2) per- 
formed at higher-than-chance levels in 
discriminating the location of punctate 
tactile stimuli from different combinations 
of three skin sites. (b) When the results 
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putational goal (in this case identifying the stimulus location) 
by using different encoding strategies. Thus, coding of stimulus 
location in the SII cortex could use the temporal structure of its 
neuronal population sensory responses, whereas areas 3b and 2 


al ensemble firing (on the order of 10-15 ms). Because bin 
clumping did not affect the trial classification accomplished by 
using area 3b and 2 neurons, these results raise the hypothesis 
that ensembles of cortical neurons may achieve the same com- 
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Fig. 8. For all three cortical areas analyzed (3b, Sll and 2), the single-trial discrimination capability varied as a function of the ensemble size. 
Shown here are the effects of sequential removal of the best predictor neurons from the population (neuron dropping) on single-trial dis- 
crimination of the stimulus location. (a) Neuron dropping in two SII ensembles (from monkeys 2 and 3) resulted in similar trends of decay in 
single-trial discrimination ability (percent correct). Notice that when the SII ensembles were reduced to about five neurons, the performance 
was close to chance (33%). (b) Neuron dropping in areas 3b and 2. The curves depict the discrimination ability as a function of neural ensem- 
ble size for these areas. Graceful decay in discrimination performance also occurred as a result of decreasing ensemble size. 
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Fig. 9. The effect of bin clumping on the discrimination capability of cortical areas. Whereas a significant reduction in discrimination capabil- 
ity was observed when bin clumping (that is, increasing the size of the bin describing neuronal firing from 3 to 45 ms) was applied to SII neural 
ensembles in monkeys 2 (a) and 3 (b), no effect was observed either in area 3b (c, monkey 2) or 2 (d, monkey 3) neural ensembles. These 
results suggest that the temporal structure of population responses could be more fundamental for encoding the location of a tactile stimu- 


lus in area SII than in the other two somatosensory cortical areas. 


could encode the same stimulus locations by simply using varia- 
tions in the mean ensemble firing rate. Moreover, our results sug- 
gest that SII contains the requisite information for coexistence 
of both temporal and rate coding schemes because above-chance 
discrimination was observed for the SII cortex of all three mon- 
keys after the temporal structure of neuronal sensory responses 
were eliminated by bin clumping. Therefore, a residual rate code 
could still be sufficient for ensembles of SII neurons to compute 
the location of a tactile stimulus on a single trial. 

As described above (Fig. 5b), distributions depicting the min- 
imal response latency for neurons located in three parietal corti- 
cal areas showed a great deal of overlap at about 10 to 15 ms after 
the stimulus delivery. However, neurons located in all cortical 
areas also exhibited long-latency responses (15-45 ms). By inde- 
pendently testing the contribution of three consecutive 15 ms bins 
(0-15, 15-30 and 30-45 ms) on the discrimination capability of 
each cortical area, we observed that the highest single-trial dis- 
crimination performance was obtained when we analyzed neural 
responses in the interval from 0 to 30 ms in areas 3b and SII and 
in the interval from 15 to 45 ms for area 2. These results provided 
further support for the notion that the many cortical areas that 
constitute the primate somatosensory cortex could operate almost 
simultaneously to compute the location of a given stimulus. 


Discussion 

The results described here demonstrate that the spatiotemporal 
firing patterns of neural ensembles, formed by broadly-tuned 
neurons and located in three cortical areas of the primate 
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somatosensory cortex, contain enough information to specify 
the location of punctate tactile stimuli on a single trial. This study 
also provides the first compelling evidence that cortical areas 
could carry out such computations almost simultaneously, but 
using different representational strategies (that is, temporal ver- 
sus rate coding or a combination of both schemes) to achieve the 
same goal. In particular, the spatiotemporal character of neu- 
ronal responses in the SII cortex was shown to contain the req- 
uisite information for the encoding of stimulus location using 
temporally patterned spike sequences, whereas the simultane- 
ously recorded neuronal responses in areas 3b and 2 contained 
the requisite information for rate coding. 

Based on our results, we hypothesize that information-pro- 
cessing strategies may vary across cortical areas of the primate 
somatosensory system but that these areas may still act collec- 
tively to encode a variety of stimulus attributes. In support of 
this possibility, processing of sensory information in the primate 
brain involves interactions between dozens of cortical areas, each 
of which is interconnected with a number of other areas!?°-?8, 
In the last few years, this distributed connectivity scheme has 
provided the basis for a new theoretical framework for cortical 
function, which departs from the notion that each cortical area is 
dedicated to one specific task and contains highly specialized, 
feature-detector neurons. The alternative notion suggests that 
single cortical areas, and the neurons within them, participate in 
a variety of functions and that they interact extensively-!?. This 
model of cortical function incorporates the benefits of popula- 
tion coding in distributed networks?! and implies that a large 
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number of neurons is involved in any particular perceptual or 
motor task®!!?!, Therefore, the fundamental focus of cortical 
computation shifts from the single neuron toward interconnect- 
ed neural ensembles as originally proposed by Hebb®”?. 

Therefore we envision that neural populations located in dif- 
ferent cortical areas could either share the same type of infor- 
mation or cooperate to improve their individual performance. 
Moreover, the finding that information about the location of a 
tactile stimulus could be readily extracted from all three cortical 
areas suggests that these fields may collaborate in the definition of 
a unified perceptual experience of the stimulus location. Cortical 
areas could also use the information about stimulus location to 
produce unique computations or to relate their outputs to a pre- 
cise representation of the stimulus location. In a more general 
sense, our results raise the possibility that cortical areas, although 
specialized to differing degrees, could process multiple attribut- 
es of a sensory stimulus simultaneously. Thus, by using combi- 
nations of encoding mechanisms (temporal versus rate coding), 
populations of neurons could ‘multiplex’ various types of infor- 
mation in their firing patterns*®*!. According to this scheme, sin- 
gle neurons would likely participate in multiple coexistent 
representations within each cortical area. Paradoxically, howev- 
er, their individual firing patterns would not reliably encode any 
of the stimulus features on a single trial. We acknowledge that, 
in this model of cortical function, there is no strict requirement 
for a specific binding mechanism, such as synchronous neuronal 
activity”, although such a mechanism may be involved in other 
aspects of sensorimotor processing besides identifying stimulus 
location. In our model, information regarding the tactile stimu- 
lus location would reach all somatosensory cortical areas within 
a relatively small time interval, and consequently, there would be 
no need to ‘bind’ this type of information together. 


Methods 

SURGICAL AND ELECTROPHYSIOLOGICAL PROCEDURES. In each animal, mul- 
tiple microelectrode arrays (NBLABS, Dennison, Texas), each contain- 
ing 16 teflon-coated, stainless steel microwires (50 4m in diameter) 
forming a 2 X 8 matrix (total sampling area, 2 mm?), were chronically 
implanted in different cortical areas under general anesthesia (15 mg/kg 
ketamine, 5mg/kg xylazine and 1% halothane). During the surgical pro- 
cedure for microwire array implantation, the receptive fields of single 
and multiple units were characterized to ensure correct placement of 
each array. Five to seven days after the surgical procedure, animals were 
seated in a recording chamber. A 96 channel many neuron acquisition 
processor (MNAP, Plexon, Dallas, Texas) was used to acquire and dis- 
criminate single neural unit activity from each of the implanted 
microwires. Time-amplitude discriminators and a modified version'® 
of a principal component algorithm*? were used to isolate single corti- 
cal units in real time. Analog samples of the waveforms of cortical poten- 
tials and the time of occurrence of each spike were stored. Off-line 
analyses of interspike interval histograms, autocorrelograms and wave- 
form shapes were used to crossvalidate the on-line discrimination of 
single units. Once single units had been discriminated, punctate tactile 
stimulation (step pulses with 100 ms duration delivered at 1Hz) were 
delivered to different locations of the hand, forearm and face, using a 
computer-controlled vibromechanical probe driven by a Grass S8800 
stimulator. A total of 360 trials were delivered at each skin site in a ran- 
dom sequence. Up to 25 skin sites were stimulated in each animal. Tac- 
tile stimulation was delivered in both awake and lightly anesthetized 
(ketamine i.m.) animals. The results obtained were virtually identical 
in both conditions. 


SINGLE-TRIAL ANALYSIS OF NEURAL ENSEMBLE DATA. Three pattern-recogni- 
tion techniques were used to measure the ability of neural ensembles 
located in three somatosensory cortical areas (3b, SII and area 2) to 
encode stimulus location on a single trial: linear discriminant analysis 
(LDA), a learning vector quantization (LVQ) artificial neural network 
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(ANN), and a back-propagation (BP) ANN. Our procedures for apply- 
ing LDA, a classical multivariate statistical technique, have been 
described*>. The BP ANN was chosen because it is widely used for non- 
linear pattern recognition, not only in computer science and engineer- 
ing™ but also in neuroscience, where it has been used in a variety of 
studies? 53, The LVQ ANN was chosen because it provides a nonpara- 
metric technique for classifying large and sparse non-linear pattern vec- 
tors*’. This made the LVQ ANN an attractive option for single-trial 
classification of input patterns from simultaneous multi-neuron record- 
ings. A multiprocessor pentium-based workstation running MATLAB 
(Mathworks, Boston, Massachusetts) and its statistics and neural net- 
work toolboxes was used for all pattern recognition analysis. We used a 
modified LVQ ANN to apply the Kohonen’s optimized-learning-vector 
quantization algorithm, which is a supervised version of Kohonen’s self- 
organizing feature map*”. The network included two hidden units and 
one output unit per class (i.e. stimulus site) to be discriminated. Thus, 
when data from three locations were used, the networks had 6 hidden 
units and 3 output units. The number of training epochs used in these 
analyses were the product of the number of hidden units and the num- 
ber of training trials. The feedforward BP ANN was configured as a 
supervised learning technique that produced parametric outputs in prob- 
abilistic ranges. It used two hyperbolic tangent hidden units per class 
(skin site) and one log sigmoid output unit per class, and trained using 
the resilient BP learning rule. The number of training epochs used was 
determined by ‘early stopping™*, with increments of 100 epochs. 
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The exquisite modular anatomy of the rat somatosensory sys- 
tem makes it an excellent model to test the potential coding 
strategies used to discriminate the location of a tactile stimulus. 
Here, we investigated how ensembles of simultaneously re- 
corded single neurons in layer V of primary somatosensory (Sl) 
cortex and in the ventral posterior medial (VPM) nucleus of the 
thalamus of the anesthetized rat may encode the location of a 
single whisker stimulus on a single trial basis. An artificial neural 
network based on a learning vector quantization algorithm, was 
used to identify putative coding mechanisms. Our data suggest 
that these neural ensembles may rely on a distributed coding 
scheme to represent the location of single whisker stimuli. 
Within this scheme, the temporal modulation of neural ensem- 
ble firing rate, as well as the temporal interactions between 


neurons, contributed significantly to the representation of stim- 
ulus location. The relative contribution of these temporal codes 
increased with the number of whiskers that the ensembles must 
discriminate among. Our results also indicated that the Sl 
cortex and the VPM nucleus may function as a single entity to 
encode stimulus location. Overall, our data suggest that the 
representation of somatosensory features in the rat trigeminal 
system may arise from the interactions of neurons within and 
between the SI cortex and VPM nucleus. Furthermore, multiple 
coding strategies may be used simultaneously to represent the 
location of tactile stimuli. 

Key words: population coding; multi-electrode; temporal 
code; ventral posterior medial nucleus; barrel cortex; primary 
somatosensory cortex 


Rodents actively use their facial whiskers to explore their envi- 
ronment. Removal of these whiskers results in impaired perfor- 
mance on various tactile discrimination tasks (Vincent, 1912; 
Schiffman et al., 1970; Brecht et al., 1997). These behavioral 
experiments have underscored the importance of the facial whis- 
kers in determining the spatial location of tactile stimuli. For 
example, by clipping the large caudal vibrissae of the rat’s whisker 
pad and comparing performance of rats on spatial versus object 
recognition tasks, Brecht et al. (1997) demonstrated that these 
large caudal whiskers were critically involved in spatial tasks but 
not in object recognition tasks. These authors (Brecht et al., 1997) 
suggested that these whiskers act as “distance decoders,” the 
function of which is to determine the location of obstacles and 
openings. 

At the neural level, experimental lesions within the “whisker 
area” of the rat somatosensory system support the hypothesis that 
the caudal whiskers and their associated neural pathways are 
necessary for spatial discrimination (Hutson and Masterton, 
1986). It is unclear, however, how neurons may encode the spatial 
location of tactile stimuli. Coding mechanisms for determining 
the spatial location of a stimulus in sensory space generally fall 
into two categories: local versus distributed coding. In the local 
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coding scheme, the sensory space is divided into nonoverlapping 
areas that can be resolved by small groups of topographically 
arranged neurons. These neurons necessarily have small recep- 
tive fields. One of the potential benefits of topographic maps in 
sensory systems is the ability to easily identify the location of a 
stimulus: localized groups of neurons respond specifically to the 
presence of a stimulus in a restricted portion of the sensory space, 
whereas the other neurons are quiescent. Thus, local coding can 
offer exquisite specificity and speed in behavioral response. How- 
ever, lesions of a particular region of this map would render the 
system unable to identify stimuli delivered to discrete locations 
on the receptor organ. Conversely, in the distributed coding 
scheme, neurons have relatively large and overlapping receptive 
fields compared with the sensory resolution measured behavior- 
ally. Distributed representations allow neurons to be computa- 
tionally flexible (neurons can participate in many different aspects 
of sensory processing) (Richmond and Optican, 1987; Victor and 
Purpura, 1996) and resistant to both central and peripheral injury 
(Nicolelis, 1997). 

The specialized structure of the rodent somatosensory pathway 
is well suited to testing which of these potential coding strategies, 
local versus distributed, is used to identify the spatial location of 
sensory stimuli. The rat trigeminal somatosensory pathway con- 
sists of topographically arranged clusters (or modules) of neu- 
rons: “barrel columns” in the cortex (Woolsey and Van der Loos, 
1970; Killackey, 1973), “barreloids” in the thalamus (Van der 
Loos, 1976), and “barrelettes” in the brainstem (Ma, 1991). Each 
module corresponds isomorphically to a single whisker on the 
snout. The modular and topographic anatomy suggests that this 
system may use a local coding scheme, whereby each module 
encodes the spatial location of a single caudal whisker (Nelson 
and Bower, 1990). However, the discrete cytoarchitecture of this 
pathway stands in contrast to what is known regarding the phys- 
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iology of this system. Neurons in these structures have large 
receptive fields that extend well beyond a single caudal whisker 
(Simons, 1978; Chapin, 1986; Armstrong-James and Fox, 1987; 
Simons and Carvell, 1989; Nicolelis and Chapin, 1994; Moore 
and Nelson, 1998; Zhu and Connors, 1999; Ghazanfar and 
Nicolelis, 1999). In addition, functional studies have shown in 
primary somatosensory (SI) cortex (Kleinfeld and Delaney, 1996; 
Masino and Frostig, 1996; Peterson et al., 1998; Sheth et al., 
1998), ventral posterior medial (VPM) nucleus (Nicolelis and 
Chapin, 1994; Ghazanfar and Nicolelis, 1997), and SpV (Nicolelis 
et al., 1995) that stimulation of individual whiskers results in 
responses that extend well beyond a single barrel cortical column 
or VPM nucleus barreloid or spinal trigeminal nucleus (SpV) 
barrelette. 

To date, no study has explored the potential coding strategies in 
an anatomically modular and topographic sensory system used to 
represent the location of a stimulus on a single trial basis. The 
presumption has always been that such systems use “local” coding 
schemes to encode stimulus location. As a first step to address this 
issue, we tested the potential coding strategies of simultaneously 
recorded ensembles of single neurons distributed across layer V 
of SI cortex and VPM nucleus of the thalamus in the anesthetized 
rat and investigated how these two structures may interact with 
each other to encode the location of simple tactile stimuli. 


MATERIALS AND METHODS 


Animals and surgical procedures 


Nine adult female Long-Evans rats (250-300 gm) were used in these 
experiments. Details of surgical procedures have been described else- 
where (Nicolelis et al., 1997a). Briefly, animals were anesthetized with 
intraperitoneal injections of sodium pentobarbital (Nembutal, 50 mg/kg) 
and transferred to a stereotaxic apparatus. When necessary, small sup- 
plementary injections of sodium pentobarbital (~0.1 cc) were adminis- 
tered to maintain anesthesia during the surgery. After retraction of the 
skin and soft tissue, small, rectangular craniotomies were made over the 
SI barrel cortex and/or the VPM nucleus of the thalamus using stereo- 
taxic coordinates. For layer V cortical implants, stainless-steel microwire 
arrays (NB Labs, Dennison, TX) consisting of two rows, separated by 1 
mm, of eight microwires were used. Each microwire was Teflon-coated 
and had a 50 wm tip diameter. The inter-microwire distance within a row 
was 200 wm. For VPM nucleus implants, two bundles of eight microw- 
ires, cut at two different lengths, were used. The distance between 
bundles was ~1 mm. For all animals, we successfully targeted the 
representation for the large, caudal whiskers (B1—-4, C1l-4, D1-4, and 
E1-4) [see Ghazanfar and Nicolelis (1999) for details on identifying 
target locations]. On proper placement, microwire implants were ce- 
mented to the animal’s skull with dental acrylic. The location of all 
microwires was assessed by qualitative receptive field mapping during 
surgical implantation, later confirmed by the quantitative response pro- 
files of neurons, and then postmortem by light microscopic analysis of 
Nissl-stained sections. 


Data acquisition 

Spike sorting 

After a recovery period of 5-7 d, animals were anesthetized with sodium 
pentobarbital (50 mg/kg) and transferred to a recording chamber where 
all experiments were performed. A head stage was used to connect the 
chronically implanted microwires to a preamplifier whose outputs were 
sent to a Multi-Neuronal Acquisition Processor (Plexon Inc., Dallas, TX) 
for on-line multi-channel spike sorting and acquisition (sampling rate = 
40 kHz per channel). A maximum of four extracellular single units per 
microwire could be discriminated in real time using time-voltage win- 
dows and a principal component-based spike sorting algorithm (Abeles 
and Goldstein, 1977; Nicolelis and Chapin, 1994). Previous studies have 
revealed that under our experimental conditions, ~80% of the microw- 
ires yield stable single units and an average of 2.3 single units can be well 
discriminated per microwire (Nicolelis et al., 1997a). Examples of wave- 
forms and further details regarding acquisition hardware and spike 
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sorting can be found elsewhere (Nicolelis and Chapin, 1994; Nicolelis et 
al., 1997a). 


Recording session and whisker stimulation 


After spike sorting, the simultaneous extracellular activity of all well 
isolated single units was recorded throughout the duration of all stimu- 
lation experiments. A computer-controlled vibromechanical probe was 
used to deliver innocuous mechanical stimulation to single whiskers on 
the mystacial pad contralateral to the microwire array implant. The 
independent stimulation of 16 whiskers was performed per recording 
session per animal. Three hundred sixty trials were obtained per stimu- 
lated whisker, and the probe was then moved to another whisker (in 
random order). Whiskers were stimulated by positioning the probe just 
beneath an individual whisker, ~5—10 mm away from the skin. Extreme 
care was taken to ensure that only a single whisker was being stimulated 
at all times. A step-pulse (100 msec in duration) delivered at 1 Hz by a 
Grass 8800 stimulator was used to drive the vibromechanical probe. The 
output of the stimulator was calibrated to produce a ~0.5 mm upward 
deflection of whiskers. Stable levels of anesthesia were maintained by 
small supplemental injections of pentobarbital (~0.05 cc) and monitored 
through regular inspection of brain activity, breathing rates, and tail- 
pinch responses. 


Data analysis 
Firing rate and minimal latencies 


The minimal spike latency and the average evoked firing rate of each 
neuron were estimated using poststimulus time histograms (PSTHs) and 
cumulative frequency histograms (CFHs). CFHs were used to measure 
the statistical significance of sensory responses to tactile stimuli. These 
histograms depict the cumulative poststimulus deviations from prestimu- 
lus average firing seen in the PSTHs. In other words, the CF Hs describe 
the probability that the cumulative frequency distribution in the histo- 
gram differs from a random distribution, as computed by a one-way 
Kolmogorov—Smirnov test. Neuronal responses were considered statis- 
tically significant if the corresponding CFH indicated a p < 0.01. These 
analyses were performed on commercially available software (Stranger, 
Biographics, Winston-Salem, NC). For CFHs of statistically significant 
responses, the minimal latencies were measured using a single neuron 
analysis program based on Kernel Density Estimation and written in 
Matlab (Mathworks, Natick, MA) by Mark Laubach and Marshall Shuler 
(MacPherson and Aldridge, 1979; Richmond and Optican, 1987; Ghaza- 
nfar and Nicolelis, 1999). Details of this analysis procedure have been 
reported elsewhere (Ghazanfar and Nicolelis, 1999). 


Population histograms 


Population histograms describe the sensory response of simultaneously 
recorded neural ensemble to the deflection of a single whisker as a 
function of poststimulus time. These three-dimensional plots are essen- 
tially a collection of single neuron PSTHs stacked next to each other. 
These can be generated using a range of bin widths (1, 3, 6, 10, 20, and 
40 msec). The x-axis of these plots represents poststimulus time in 
milliseconds, the y-axis represents the neuron number, and the z-axis 
represents response magnitude in spikes per second. The neurons are 
arranged randomly along the y-axis. 


Single trial analysis of neural ensemble firing patterns 


Extracting information from the firing patterns of populations of neurons 
is difficult largely because of the combinatorial complexity of the problem 
and the uncertainty about how information is encoded in the nervous 
system. Our previous studies indicated that a large number of neurons 
are active in the rat thalamocortical loop after the deflection of a single 
whisker (Nicolelis and Chapin, 1994; Ghazanfar and Nicolelis, 1997; 
Nicolelis et al., 1997a). At spike-to-spike resolution, there is also a high 
degree of variability in the spike train of an individual neuron (Shadlen 
and Newsome, 1998; A. Ghazanfar and M. Nicolelis, unpublished obser- 
vations). Although both the number of spikes produced by a neuron and 
their timing may vary from trial to trial, at the neural ensemble level the 
location of a stimulus may be identified in a statistically predictable 
manner. Pattern recognition approaches using multivariate statistical 
methods, such as linear discriminant analysis, and artificial neural net- 
works (ANNs) are effective tools for investigating this possibility (Dead- 
wyler and Hampson, 1997; Nicolelis et al., 1999). 
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Figure 1. Statistical pattern recognition using an artificial neural network 
(ANN). The ANN used a nearest-neighbor classifier algorithm, learning 
vector quantization, to classify our large, sparse neural ensemble datasets. 
The system was a multilayered, feedforward ANN with full connectivity. 
In this case, the transfer function (F,) is a Euclidean distance measure. 
The first layer consisted of our raw data; the second layer contained two 
artificial neural units (AN Us) for each class (i.e., the number of whiskers); 
the third layer had the same number of AN Us as classes; and the fourth 
layer (data not shown) was the output layer. 


Artificial neural network based on the learning vector 
quantization classifier 


In this study, an ANN was used for statistical pattern recognition analysis 
of the thalamocortical responses to tactile stimuli on a single trial basis. 
The ANN was constructed in Matlab using an optimized learning vector 
quantization (LVQ) algorithm (Kohonen, 1997). The LVQ ANN is a 
nearest-neighbor classifier, which provides a nonparametric technique 
for classifying large and sparse nonlinear pattern vectors. The LVQ 
algorithm was selected because of its simplicity of design and its ability to 
handle our extremely large and sparse neural ensemble data sets. By 
using this approach, preprocessing of neural ensemble data, by using 
principal or independent component analyses, was not necessary as a 
primary step in the analysis (Nicolelis et al., 1998). Thus, the only 
parameters available to the LVQ ANN for pattern recognition were the 
firing rate and the temporal patterning of neuronal firing within simul- 
taneously recorded thalamocortical ensembles. 

In the Appendix, we have described in detail the logical structure and 
mathematical basis for an LVQ algorithm-based ANN used in our study. 
Here, we will briefly review the specific parameters used in our study. 
The first layer defined the input layer and consisted of our raw spike train 
data (Fig. 1). The second layer contained two artificial neural units 
(ANUs) for each class (i.e., the number of stimulus locations—whis- 
kers—to be discriminated). The output value of each second layer ANU 
was determined by an Euclidean distance function. The third layer of the 
ANN had the same number of units as the number of classes (i.e., the 
number of stimulus locations). A value of 1 was assigned to the third layer 
ANUs corresponding to the “winning” second layer ANU, whereas the 
rest of the third layer units were assigned the value of 0. Thus, if the fifth 
ANU in the second layer had the greatest output value, the fifth AN U of 
the third layer would output 1, whereas the rest of the AN Us in that layer 
would output 0 sec. Each ANU in this final layer represented a unique 
subset of second layer AN Us. The fourth and final (output) layer of the 
ANN also contained the same number of units as there were classes of 
stimulus sites to be discriminated. 

The analysis of our data set included two phases: training and testing. 
During the training phase of the analysis, the ANN searched for patterns 
closest in Euclidean distance to one of the weight vectors. For every 
analysis in this study, 25% of the trials were used for training the LVQ 
ANN, and the remaining 75% were used as testing trials. To obtain an 
accurate assessment of the network performance in classifying our neu- 
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ral ensemble data, four-way cross-validation was used and provided us 
with a measurement of error. Thus, all trials used for training in one 
session were then used in an independent session for testing and vice 
versa. The spike train of each neuron in the ensemble from 0 to 40 msec 
poststimulus time was used in all analyses. Unless noted otherwise, 4 
msec bins were used to define the contribution of each neuron to the 
ensemble input vector. 


Exploring putative coding mechanisms 


Our basic approach to investigating coding mechanisms was to compare 
statistical pattern recognition performance by the ANN using normal, 
“raw” neural ensemble data versus different manipulations of that data 
set. Raw data manipulations included removing the number of neurons 
within an ensemble, reducing the temporal resolution of spike trains, and 
disrupting the phase relationships or correlated activity between 
neurons. 

Local versus distributed coding. One of the hallmarks of distributed 
coding is the graceful degradation of ensemble performance after the 
removal of neurons from an ensemble. We tested this by measuring 
ensemble performance on discriminating four different whiskers (B1, B4, 
El, and E4; chance performance = 25%) and then removing the best 
predictor neuron from the ensemble one at a time, sequentially. The best 
predictor neuron was determined by running the analysis with each 
neuron taken out in turn and then finding the neuron that had the most 
detrimental effect on ensemble performance when removed. This neuron 
was then defined as the “best predictor” neuron of the ensemble. Once 
this neuron was removed, the analysis was run again to quantify the 
performance of the ensemble without that neuron and find the next best 
predictor neuron. 

Because we found that the effect of removing neurons from ensembles 
resulted in a smooth degradation of ensemble performance (see Results), 
we were able to estimate the number of neurons needed to achieve a 99% 
correct level of performance by using a power function (Carpenter et al., 
1999): x = log,o(z)/log,o(y), x*w = number of neurons needed, where z 
equals the desired residual information (0.01, for 99% correct) and y 
equals the obtained residual for w neurons actually recorded. A power 
function is necessary because information capacity changes nonlinearly 
with an increasing number of neurons. 

Temporal structure of ensemble firing rate. To determine whether the 
temporal modulation of ensemble firing rates contributed to the perfor- 
mance of thalamocortical ensembles, the integration time used to de- 
scribe the sensory response (i.e., bin size) of each neuron was systemat- 
ically varied between 1 and 40 msec. Increasing the bin size degrades the 
temporal resolution of the response, allowing an assessment of the 
relative contributions of rate and temporal coding to ensemble 
performance. 

Correlated activity across neuronal spike trains: spike-shifting testing 
trials. To explore the role of covariance structure on ensemble perfor- 
mance, linear discriminant analysis (LDA) (Tabachnick and Fidell, 1996) 
was used in our analyses. LDA was used to identify sources of variance 
and to measure the covariance of firing rate activity among simulta- 
neously recorded neurons. Such sources of covariance have been sug- 
gested to be important for the coding of sensory stimuli (Nicolelis et al., 
1997b) and behavioral events (Deadwyler et al., 1996). Another useful 
characteristic of LDA is that it derives classification functions for trial- 
by-trial discrimination between different experimental groups (in this 
case, the different stimulation sites). This statistical technique has been 
used extensively for neural ensemble data analysis (Gochin et al., 1994; 
Schoenbaum and Eichenbaum, 1995; Deadwyler et al., 1996; Nicolelis et 
al., 1997b). Our application of LDA to somatosensory neural data sets 
has been described in detail elsewhere (Nicolelis et al., 1997b). 

To apply LDA and test the role of correlated activity among neuronal 
firing patterns in the performance of our ensembles, raw spike trains 
within an ensemble were temporally shifted in one of two ways. In the 
first method, “spike shifting,” the spike trains within ensemble responses 
were shifted relative to each other in random order, between +6 and 12 
msec. Thus, from a single trial’s ensemble response, the spike train of 
neuron | may have been shifted +7 msec, that of neuron 2 may have been 
shifted —10 msec, so on and so forth. This was done for every trial in the 
testing phases of the analyses. The range of “shift” times was selected 
based on the finding that neural ensemble performance significantly 
degraded only with bin sizes >6 msec for both SI cortex and VPM 
nucleus (see Results, Fig. 5B). The second method, “trial-shuffling,” was 
used to randomly replace the spike trains of each neuron with those from 
another trial. After either of these procedures, normal and shifted 
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Figure 2. The spatiotemporal responses of the same ensemble to different whiskers. A, A series of population histograms depict the spatiotemporally 
complex responses of a single SI cortical ensemble to three different whiskers (D2, B2, and E4). The x-axis represents poststimulus time (in milliseconds), 
the y-axis represents the neurons in the ensemble, and the z-axis represents firing rate (spikes per second). Each whisker elicits a unique spatiotemporal 
profile of ensemble activity. B, The minimal latency (x-axis) and firing rate (y-axis) of each neuron in the ensemble responses depicted in A are plotted 
against each other. The location of single neurons within this two-dimensional “activity field” changes as a function of stimulus location. Note that 
although there are 25 neurons in A, there are only 24 in these plots. This is because one neuron did not respond significantly to these three whiskers. 


ensemble spike trains were normalized and preprocessed using principal 
component analysis (PCA). PCA reduced the dimensions of the data set, 
a necessary step for the proper application of LDA (as opposed to LVQ) 
given the large number of variables and trials used in this study. The first 
15 principal components calculated for each trial were used as the input 
matrix for this analysis. Training and testing trials were divided the same 
as in the LVQ ANN analyses with four-way cross-validation, and dis- 
criminant functions were derived by training on normal data and testing 
with “shifted” data. Ensemble performance using LDA on principal 
components and LVQ on raw data were statistically identical (Nicolelis 
et al., 1998). 

In another set of experiments, the spike shifting procedure was applied 
to both the train and testing trials for the LDA. This prevented the LDA 
from building a statistical model using the normal data with temporal 
relations intact and then testing with spike-shifted data. Only information 
contained in the temporal modulation of firing rate for each neuron 
independently within an ensemble was available, whereas any informa- 
tion contained in the relationships between neurons was eliminated. 


Histology 


The location of each microwire was confirmed for every animal through 
examination of Nissl-stained sections. After completion of recording 
sessions, animals were deeply anesthetized with a lethal dose of pento- 
barbital and then perfused intracardially with 0.9% saline solution fol- 
lowed by 4% formalin in 0.9% saline. Brains were post-fixed for a 
minimum of 24 hr in the same fixative solution. Coronal sections of the 
whole brain (80 4m) were cut on a freezing microtome. Sections were 
then counterstained for Nissl. Microwire tracks and tip positions were 
located using a light microscope. Because microwires were chronically 
implanted and remained in the brain for several days, electrode tracks 
and tip positions could be readily identified by glial scars, obviating the 
need for electrical lesions. 


RESULTS 


Ensembles of well-isolated single neurons were recorded in the SI 
cortex and/or VPM thalamus from nine animals (SI cortex alone, 
n = 3; VPM nucleus alone, n = 3; SI cortex and VPM nucleus, 
n = 3). The average SI cortical ensemble size was 34 neurons 
(range = 26—46 neurons), whereas the average VPM ensemble 
was 31 neurons (range = 26-35 neurons). Animals with both 
cortical and thalamic implants were analyzed separately from 
those with single implants; these dual-implanted animals had an 
average of 38 neurons for cortical implants and 43 neurons for 
VPM implants. 

Using our experimental approach, the average receptive field 
(RF) size for SI cortical neurons has been estimated to be 8.5 
whiskers (Ghazanfar and Nicolelis, 1999), and the average RF 
size for VPM neurons is 13.7 whiskers (Nicolelis and Chapin, 
1994). Stimulation of a given single whisker can elicit a spatio- 
temporally complex response from a large extent of both SI cortex 
and VPM thalamus. Figure 2A depicts population histograms, 
which illustrate that the stimulation of different single whiskers 
can elicit unique large-scale distributed responses within the same 
ensemble of simultaneously recorded SI cortical neurons. Analy- 
ses of these responses (Fig. 2B) illustrate that the ensemble 
response to a single whisker stimulus was characterized by a 
unique distribution of individual neuron firing rate and minimal 
latency. Similar results have been reported for VPM ensembles 
(Nicolelis and Chapin, 1994). Can such spatiotemporal neural 
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activity patterns be used to identify the location of a tactile 
stimulus on a single trial basis? 


Comparison of single neuron versus neural ensemble 
performance discriminating among 16 whiskers on a 
single trial basis 

Using the LVQ-based ANN, the ability of single cortical or 
thalamic neurons to correctly identify the location of 1 out of 16 
whisker possibilities on a single trial basis was tested. This 1 out 
of 16 whisker discrimination set was used in many of our analyses 
because it circumvented the animal-to-animal variation in the 
placement of our microwire arrays or bundles. 

Figure 3A shows that single SI cortical neurons (black bars, n = 
60) correctly classified the tactile location on 8.68 + 1.9% of the 
trials (mean + SD; range = 6.06-13.07%), whereas single VPM 
neurons (gray bars, n = 63) performed significantly better (un- 
paired ¢ test, t(;51) = —2.34, p < 0.05), correctly classifying on 
average 10.85 + 6.95% of trials (mean + SD; range = 5.69- 
39.54%). Because chance performance for 1 out of 16 whiskers 
was 6.25%, individual neurons performed slightly above chance. 

We next investigated how much neural ensembles outper- 
formed single neurons. Figure 3B shows that SI cortical ensem- 
bles (average ensemble size = 34 neurons) performed correctly 
on 46.36 + 3.55% (mean + SD; range = 40.91-51.42%) of trials, 
and VPM neural ensembles (average ensemble size = 31 neu- 
rons) performed correctly on 68.34 + 12.23% (mean + SD; 
range = 50.73—81.66%) of trials. On average, SI cortical ensem- 
bles performed 7.4 times better than chance and 5.3 times better 
than the average single cortical neuron. Along the same lines, 
VPM ensembles performed 10.9 times better than chance and 6.3 
times better than the average single VPM neuron. Overall, VPM 
ensembles performed significantly better than SI cortical ensem- 
bles (unpaired f test, f(22) = —4.33, p < 0.0005). These results 
indicated that although the firing patterns of single neurons in the 
thalamocortical pathway can be used to identify the location of a 
single whisker slightly above chance levels, small neural ensem- 
bles could perform several times better than chance and several 
times better than even the best single neurons. 


Graceful degradation of ensemble performance 

We tested whether SI cortical and VPM ensembles exhibited 
graceful degradation in performance after the sequential removal 
of “best predictor neurons” (see Materials and Methods). Figure 
4A shows the results of this analysis for two comparably sized 
neural ensembles from SI cortex (top panel) and VPM (bottom 
panel ). Both the SI cortical and VPM neural ensembles exhibited 
a smooth (“graceful”) degradation of performance on discrimi- 
nating among four different whiskers (B1, El, B4, and E4). 
Similar curves were seen for all other animals (data not shown). 
In most cases, there was a smooth decay in ensemble perfor- 
mance, and chance performance was not reached until only a few 
neurons remained in each of the ensembles. Notice that although 
individual neurons were sampled from multiple barrel cortical 
columns (Ghazanfar and Nicolelis, 1999) or barreloids (Nicolelis 
and Chapin, 1994), their contribution to the discrimination of 
different, nonisomorphic whiskers was significant. If the neurons 
were only local feature detectors, then one would expect to see 
sharp drops in ensemble performance as the neurons dedicated to 
a particular whisker were removed. These results suggest that 
despite their anatomically modular organization, the functional 
organization of the thalamocortical pathway in rats is one of a 
highly distributed system, at least for the encoding of the location 
of punctate tactile stimuli by layer V cortical and VPM neurons. 
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Figure 3. The discrimination capability of single neurons versus neural 
ensembles. A, Single SI cortical and VPM neurons were tested on their 
ability to discriminate the location of a whisker stimulus among 16 
different possibilities. The x-axis represents the percentage of correct 
classified trials, and the y-axis represents the number of neurons. Black 
bars: SI cortical neurons. Gray bars: VPM neurons. Chance performance 
was 6.25%. On average, both cortical and thalamic neurons performed 
slightly above chance levels. B, Ensembles of SI cortical and VPM neu- 
rons were similarly tested on their ability to discriminate among 16 
whiskers. The percentage of correct trials is plotted on the y-axis. It can 
be seen that ensembles of neurons perform several times better than 
chance and several times better than the average single neuron. Further- 
more, V PM ensembles perform better than SI cortical ensembles. Chance 
performance was 6.25%, as indicated by the dashed line. Error bars show 
1 SEM. 


A comparison of the rate of decay in performance was made 
between cortical and thalamic ensembles. Because of the variable 
size of ensembles, this measurement was taken between n and 
n—-10, where n equals the number of neurons in the ensemble. On 
average, SI cortical ensembles (2.1% per neuron) decayed slower 
than VPM ensembles (3.4% per neuron) (unpaired t-test, t(39) = 
3.66, p < 0.001). This suggests that a smaller number of VPM 
neurons carry the information for encoding tactile stimulus loca- 
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Figure 4. Distributed coding properties of cortical and thalamic ensem- 
bles. A, Degradation of ensemble performance discriminating four dif- 
ferent whiskers (B1, B4, E1, and E4) was measured after the sequential, 
one-by-one removal of the best predictor neuron. Here it can be seen that 
the sequential removal of SI cortical neurons resulted in the graceful 
degradation of one cortical ensemble performance (top panel). All other 
cortical ensembles showed the same effect. Similarly, the performance of 
a thalamic ensemble also gracefully degraded when neurons are removed 
one by one (bottom panel ). Chance performance was 25%, as indicated by 
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tion when compared with SI cortex. This is in accordance with 
both the single neuron and ensemble data from the VPM nucleus. 

In an effort to estimate the minimal size of thalamic and 
cortical ensembles capable of discriminating the location of a 
whisker at a 99.9% level, a power equation was used to extrapo- 
late from each subject’s neural ensemble performance discrimi- 
nating 4, 8, 12 and 16 whiskers (see Fig. 6 for whisker identities). 
As expected, for both SI cortex and VPM, more neurons were 
needed to achieve 99.9% performance on increasingly difficult 
discriminations (Fig. 4B). Based on the LVQ ANN, 99% dis- 
crimination of one out of four whiskers would require on average 
129 SI neurons and 75 VPM neurons. A 1 out of 16 whisker 
discrimination would require 269 cortical and 137 thalamic neu- 
rons. For all discrimination sets, almost twice as many cortical 
neurons were needed than VPM neurons. Interestingly, the num- 
ber of neurons needed seemed to plateau between discrimina- 
tions among 1 out of 12 and 1 out of 16 whiskers for both SI cortex 
and VPM neurons—the required ensemble size did not increase 
linearly with the complexity of the discrimination. This is another 
hallmark of distributed coding. Above a given ensemble size, 
some collective property of the neuronal population could ac- 
count for this effect. Therefore, we next investigated what form 
such a collective property may take. 


Interaction between rate and temporal coding in 
cortical and thalamic neural ensembles 

To investigate how the temporal modulation of neural ensemble 
firing affected the discrimination of tactile location on a single 
trial basis, we parametrically increased the size of the integration 
window (i.e., bin size) used to generate the input vector for our 
LVQ ANN analysis. By increasing the bin sizes, we degraded the 
temporal resolution of the population signal but kept the number 
of spikes produced by the ensemble constant. This effect can be 
seen in Figure 54 where the same SI cortical ensemble response 
to a single whisker is plotted with different bin sizes (3, 6, 10, and 
20 msec). Notice that although temporal information was de- 
graded by this manipulation, simple firing rate differences could 
conceivably be used to discriminate among different whisker 
deflections. Figure 5B shows the performance of SI cortical and 
VPM ensembles when the LVQ algorithm was used to measure 
their ability to discriminate among 1 out of 16 whiskers using 
different bin sizes (1, 3, 6, 10, 20, and 40 msec bins). For both SI 
cortical and VPM ensembles, discrimination performance de- 
graded significantly when the bin size was increased to 10 msec 
(SI cortex: 6 vs 10 msec, f(15) = 4.15, p < 0.001; VPM: 6 vs 10 
msec, f(15) = 5.58, p < 0.0005). For cortical ensembles, a bin size 
of 1 msec actually degraded performance significantly (vs 3 msec 
bins, f44;, = —0.03, p < 0.0005). Further reduction in perfor- 
mance was observed as bin sizes were increased beyond 10 msec. 
Nevertheless, both SI cortical and VPM ensembles still per- 


— 


the dashed line. B, The number of neurons needed within a cortical or 
thalamic population to achieve 99.9% correct performance was extrapo- 
lated from their average performance discriminating 4, 8, 12, and 16 
whiskers. The number of cortical neurons needed to achieve near-perfect 
performance was approximately twice as much for discriminating 16 
whiskers versus 4 whiskers (black line). Similarly, the number of neurons 
necessarily increased with increasing discrimination difficulty. Interest- 
ingly, for both SI cortical and VPM ensembles, the number of neurons 
needed reached plateau between 12 and 16 whiskers, suggesting that the 
number of neurons needed is not linearly related to the number of classes 
to be discriminated. 
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Interaction between rate and temporal coding. A, A series of population histograms depict the effects of increasing bin size (3, 6, 10, and 20 


msec) on the temporal resolution of ensemble responses. The x-axis represents the number of bins, the y-axis represents the neurons, and the z-axis 
represents the response magnitude. As shown here, such temporal manipulations preserve the overall number of spikes in the response but destroy the 
temporal resolution. B, The interaction between rate and temporal coding was tested on SI cortical and VPM thalamic ensembles discriminating among 
16 whiskers. Ensemble performance was systematically tested with different bin sizes (1, 3, 6, 10, 20, and 40 msec). It can be seen that performance 
degraded significantly for both cortical and thalamic ensembles when bin sizes >6 msec were used. This suggests that the temporal distribution of spikes 
conveys important information regarding tactile stimulus location. Chance performance was 6.25%, as indicated by the dashed line. 


formed at greater than chance levels when only the overall aver- 
age firing rate within a trial was used (40 msec bins). Thus, 
although the temporal modulation of ensemble response con- 
veyed significant information about stimulus location, the total 
number of spikes seemed to contribute a larger proportion of 
information under these particular experimental conditions. 


Effects of disrupting correlated activity among neurons 
Spike shifting and trial shuffling of testing trials 

Because the activity of a large number of neurons was recorded 
simultaneously, we could test whether the covariance structure 
within ensemble responses contributed to ensemble performance. 
Because the LVQ algorithm is not particularly suited to analyzing 
the covariance structure among neurons, LDA, which explicitly 
looks for covariance structure, was applied to our ensemble data 
sets. LDA was applied before and after temporally shifting the 
ensemble spike trains +6-12 msec relative to each other (spike 
shifting) and before and after randomly shuffling the trials of 
individual neuron spike trains (trial shuffling). In these data 
manipulations, first, a set of linear discriminant functions was 
derived using normal, unshifted trials. Next, three sets of new 
trials (which were not used to derive the discriminant functions) 
were used to measure the ability of these discriminant functions 
to predict the location of a stimulus on a single trial basis. These 
three sets included (1) normal, unshifted trials, (2) trials in which 
the timing of individual spikes for each neuron had been shifted 


relatively to each other (spike shifting), and (3) a set of data in 
which the order of trials was randomized for each neuron tested 
with the shifted data set (trial shuffling). The time range for spike 
shifting was selected based on the degradation of ensemble per- 
formance when the temporal resolution was decreased from 6 to 
10 msec (Fig. 5B). To assess the possibility that correlated activity 
may play different roles when different numbers of whiskers are 
used (i.e., increasing the difficulty of the discrimination), the same 
analysis was performed on thalamic and cortical ensemble re- 
sponses to different combinations and numbers of whiskers (1 out 
of 4, 8, 12, or 16 whiskers). 

Spike shifting of single trial neuronal responses resulted in an 
overall increase in the variance calculated from mean ensemble 
responses (0.4789 vs 0.4918; f(5) = —2.18, p < 0.05). This value 
was obtained by measuring the change in variance of each trial’s 
ensemble response from the mean ensemble response before and 
after the spike shifting procedure. As result of this increase in 
variance, Figure 6 shows that in every case, for both SI cortex and 
VPM neurons, disrupting the correlated activity between neu- 
rons using the spike shifting procedure resulted in a significant 
decrease in ensemble performance, regardless of which whisker 
combination was used. In SI cortex, for example, single trial 
discrimination of the location of a stimulus in one out of four 
whiskers dropped from 72.36 + 6.91% (mean + SD) correct to 
47.17 + 5.72% (mean + SD) correct after the correlated activity 
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Figure 6. Effect of disrupting the correlated activity between neurons on ensemble performance. The effect of shifting spike trains relative to each other 
within an ensemble response was measured using linear discriminant analysis. Only spike trains of the testing trials were shifted randomly between +6 
and 12 msec. For both SI cortex (top panels) and VPM (bottom panels) neurons, ensemble performances were significantly worse after the covariance 
structure among neurons within ensembles was disrupted, regardless of the difficulty of the discrimination task (4, 8, 12, and 16 whiskers; the spatial 
patterns are depicted at the top of the figure). Chance performance is indicated by a dashed line in each graph. 


between neurons was disrupted, a difference of ~35%. Similarly, 
for VPM neurons, correct discrimination of one out of four 
whiskers dropped from 77.38 + 6.44% (mean + SD) correct to 
62.28 + 5.40% (mean + SD) correct after correlated activity. 

The magnitude of the effect of disrupting correlated activity 
varied according to the difficulty of the discrimination and 
whether the ensembles were located in SI cortex or VPM thala- 
mus. This was measured by taking the difference between normal 
and spike-shifted performance and dividing this by the value 
obtained for normal performance [i.e., (normal — spike-shifted)/ 
normal]. The effect of disrupting the temporal relationships be- 
tween neurons was greater when the location of the stimulus had 
to be identified from a larger number of whiskers for both SI 
cortex and thalamus (two-way ANOVA, Fis) = 36.93, p < 
0.000001). The same analysis revealed that correlated activity 
between neuronal firing played a larger role in cortical ensemble 
performance than thalamic ensemble performance (Fis) = 
17.23, p < 0.0001). 

Next, trial shuffling was used to evaluate whether intratrial- 
correlated activity across the neural ensemble played any role in 
coding tactile location. To measure this potential coding strategy, 
the spike trains of each neuron were randomly replaced with the 
spike train of the same neuron from another trial. Shuffling of the 
trial order for each neuron resulted in an overall decrease in the 
variance as measured by calculating the variance of each trial’s 
ensemble response relative to the mean ensemble response before 
and after trial shuffling (0.4789 vs 0.2455; f(s) = 10.57, p < 
0.00001). Figure 7 shows that for both SI cortex and VPM 
neurons, ensemble performance discriminating among one out of 
four whiskers was minimally affected (ensembles performed 
slightly better) or not at all by trial shuffling, especially when 
compared with spike shifting. Similar results were obtained for 1 


out of 8, 12, and 16 whisker sets (data not shown). Because 
shifting spike trains temporally relative to each other within a 
trial disrupted performance (but shuffling them across trials did 
not), it appears that the temporal relationships between spike 
trains within an ensemble response played a significant role in 
identifying stimulus location. 


Spike shifting both the training and testing trials 

By spike shifting both the training and testing trials in the LDA, 
we were able to prevent the analysis procedure from using any 
form of correlated activity to build its statistical model. In com- 
parison with the performance of intact cortical and thalamic 
ensembles, we found that spike shifting both the training and 
testing trials of cortical and thalamic ensembles significantly dis- 
rupted performance (Fig. 8). Unlike disrupting the test trials 
alone, the magnitude of the effect was not different for cortical 
versus thalamic ensembles (two-way ANOVA, Fi; ss) = 0.603, 
p = 0.439). However, as the number of whiskers increased in the 
discrimination set, the magnitude of the effect also increased 
(two-way ANOVA, F(3.8) = 28.58, p < 0.00001). We interpret 
these results as suggesting that correlated activity plays a signif- 
icant role in the coding of tactile location in both thalamic and 
cortical ensembles and that the role of correlated activity as a 
coding dimension increases as function of the number of whiskers 
to be discriminated. However, disrupting training and testing 
trials did not have a greater effect on coding tactile location than 
did disrupting testing trials alone. Across all whisker sets, disrupt- 
ing testing trials alone had a greater effect on both cortical and 
thalamic ensembles than spike shifting training and testing trials 
as measured by magnitude differences in performance (SI cortex: 
tar = —7.38, p < 0.00001; VPM: ti47) = —29.14, p < 0.001). 


48 


Ghazanfar et al. e Coding by Somatosensory Thalamocortical Ensembles 


& SI cortex 
0.9 vem 


° 
N 


S 
a 


Percent correct (X100%) 


o 
= 


= 
o 


Normal Trial- 
Shuffling 


Trial- 
Shuffling 


Spike- 
Shifting 


Normal Spike- 


Shifting 
Disruption Method 


Figure 7. Influence of intratrial correlations on ensemble performance. 
Another method of disrupting the covariance between neurons, trial 
shuffling, was applied to cortical and thalamic ensembles. With this 
method, the spike train neurons were randomly removed and replaced 
with spike trains from the same neurons from other trials. This disrupted 
any potential intratrial covariation. As seen in this Figure, this method did 
not degrade ensemble performance for SI cortex (gray bars) or VPM 
(black bars) neurons as measured by linear discriminant analysis. Chance 
performance was 25%, as indicated by the dashed line. 


Temporal evolution of ensemble performance 

Because the sensory responses of SI and VPM neurons exhibited 
considerable modulation over poststimulus time (Nicolelis and 
Chapin, 1994; Ghazanfar and Nicolelis, 1997, 1999), we investi- 
gated the poststimulus time course of SI cortical and VPM 
ensemble performance in discriminating among | out of 16 whis- 
kers. We did this by dividing the poststimulus activity into eight 
5 msec poststimulus time epochs. Figure 9A demonstrates that 
VPM ensemble coding was bimodal, peaking around 5-15 msec 
and then again at 25-30 msec, whereas SI cortical ensemble 
coding peaked between 10 and 20 msec. In addition, above chance 
performance occurred in both structures concurrently for several 
milliseconds. In support of this result, Figure 9B depicts the 
simultaneously recorded activity of SI cortical and VPM ensem- 
bles after the deflection of a single whisker. After an initial 
activation of VPM ensembles between 5 and 15 msec, activity was 
concurrent between SI cortex and VPM ensembles for several 
milliseconds. Cortical activity peaked between 15 and 30 msec, in 
agreement with the ensemble performance analysis. Although 
not unequivocal, this temporal pattern of ensemble performance 
is suggestive of reverberatory activity between SI cortical and 
VPM ensembles and their collective involvement in the discrim- 
ination of stimulus location. 


Discrimination performance of simultaneously 
recorded thalamocortical ensembles versus SI cortex 
or VPM ensembles alone 

To further explore the interactions between these structures, SI 
cortical and VPM ensembles were recorded simultaneously in 
three animals, and the performance of both ensembles together 
was compared with the performance of each structure alone in 
discriminating one out of four whiskers (B1, B4, El, and E4). As 
shown in Figure 10, thalamocortical ensembles performed signif- 
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Figure 8. The effect of spike shifting on both the training and testing 
trials was used to measure the influence of correlated activity in ensemble 
coding. Decorrelating activity significantly degraded ensemble perfor- 
mance for both cortical and thalamic ensembles and for all sets of 
whiskers (see Fig. 4 for whisker identities). 


icantly better than either SI cortex alone (¢,4,;) = —7.50, p < 
0.00005) or VPM alone (¢(;;) = —7.06, p < 0.0005). However, the 
increase in performance when SI cortex and VPM neurons were 
combined was not additive, suggesting that there was a degree of 
redundant information between these structures. Thus, simply 
increasing the number of neurons within a somatosensory 
thalamocortical ensemble did not necessarily increase perfor- 
mance in a linear fashion. 

In support of the contention that the SI cortex and VPM 
neurons function interactively, Figure 114 shows a plot of the 
effects of dropping the best predictor neuron from a thalamo- 
cortical ensemble of 75 neurons discriminating among four 
whiskers (Bl, B4, El, and E4), where 79% of trials were 
correctly classified. If the two structures were independent, one 
would expect to see a sharp drop in performance as the 
ensemble switches from dropping VPM neurons to SI cortical 
neurons, because VPM neurons perform, on average, better 
than SI cortical neurons (Fig. 3); instead, graceful degradation 
was still present when the two structures were combined. 
Figure 11B depicts the graceful degradation of SI cortical and 
VPM ensembles when considered separately. Again, the linear 
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Figure 9. Temporal evolution of ensemble performance. A, The time course of SI cortical (/eft) and VPM (right) ensemble performance discriminating 
among 16 whiskers was measured by dividing ensemble responses into 5 msec poststimulus time epochs (bin size within an epoch was 1 msec). SI cortical 
ensembles peaked at 10-20 msec, whereas VPM ensembles peaked at 5-15 msec and then again at 25-30 msec. Better than chance level performance 
occurred concurrently for several milliseconds between these two structures. B, Here, raw activation plots of simultaneously recorded SI cortical and 
VPM ensembles demonstrate that activity between these two structures, after the deflection of a single whisker, occurred concurrently for several 
milliseconds poststimulus time. Three-dimensional matrices were used to represent the poststimulus firing of neurons in VPM and SI neurons according 
to their location on the 2 X 8 electrode arrays implanted in each of these structures. In each electrode array (represented by two panels plotted side by 
side and separated by an empty space), the x-axis represents the mediolateral position (/eft = medial) of the neurons in the recording probe; the y-axis 
represents the rostrocaudal position (top = rostral-most wires 1 and 9; bottom = caudal-most wires 8 and 16); and the z-axis, plotted in a gray-scale 
gradient, represents the variation in neuronal response magnitude (white = higher than 4 SDs of the spontaneous firing rate; dark gray = baseline firing 
rate). All sensory responses were extracted from PSTHs obtained after 360 stimulation trials. 


sum of their performance would result in >100% correct 
classification. Extrapolation of SI cortical and VPM ensemble 
performance revealed that they would require 49 and 48 neu- 
rons, respectively, to achieve 79% correct independently, 35% 
fewer neurons than are actually needed when both structures 
were combined again. This further suggests the presence of 
redundancy in the representation of information about tactile 
stimulus location across SI cortex and VPM. 


DISCUSSION 


We found that the rat somatosensory system, despite its anatom- 
ically modular and topographic organization, could rely on a 
distributed coding scheme to represent the location of tactile 
stimuli. Within this coding scheme, both the spatial and temporal 
components of the ensemble activity conveyed significant infor- 
mation. The important temporal aspects of coding included the 
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Figure 10. Redundancy of information within thalamocortical ensem- 
bles. In animals in which SI cortex and VPM ensembles were recorded 
simultaneously, ensemble performance discriminating among four whis- 
kers (B1, B4, E1, and E4) was measured for each structure independently 
and then with the structures combined as a single ensemble. Here, 
combining structures did increase the performance of the ensemble but 
the increase was not linear, suggesting that there is a considerable amount 
of redundant information between SI cortex and VPM ensembles. 
Chance performance was 25%, as indicated by the dashed line. 


modulation of ensemble firing over poststimulus time and the 
correlated activity among neurons within ensembles. Further- 
more, we obtained evidence that suggests that the relative con- 
tribution of correlated activity among neurons in coding a stim- 
ulus location changed as a function of the discrimination set. 
Finally, we found that the thalamus and cortex can encode tactile 
information concurrently and that at least some information 
between them is redundant, suggesting that these two structures 
may function as a single unit. On the basis of these results, we 
propose that the rat somatosensory thalamocortical pathway uses 
multiple strategies to encode tactile stimulus location. The extent 
to which a strategy is used may depend on the difficulty of the task. 

Although many studies have focused on single neuron firing 
patterns (Richmond et al., 1990; McClurkin et al., 1991a; Middle- 
brooks et al., 1994, 1998; Victor and Purpura, 1996) and on 
pair-wise interactions between neurons (Gray et al., 1989; 
Ahissar et al., 1992; deCharms and Merzenich, 1996; Dan et al., 
1998), there are only few studies that have examined how ensem- 
bles of single neurons function together to represent simple 
stimuli. To circumvent this problem, some investigators have 
constructed population vectors from serially recorded single units 
to study potential coding schemes at the level of ensembles 
(Georgopoulos et al., 1986; Gochin et al., 1994; Fitzpatrick et al., 
1997). Despite its usefulness and the important insights gained 
from this approach, it does not allow one to investigate various 
“collective” strategies, such as temporal interactions. 

The recent advent of new electrophysiological techniques that 
allow one to record from populations of several tens of well 
isolated neurons simultaneously has made such studies of ensem- 
ble coding feasible (Wilson and McNaughton, 1994; Deadwyler et 
al., 1996; Nicolelis et al., 1997a, 1998). Nevertheless, identifying 
methods for analyzing such large data sets remains a challenge 
(Deadwyler and Hampson, 1997). To accomplish this goal, we 
adopted a strategy that takes advantage of pattern recognition 
algorithms that use both multivariate statistical methods (Gochin 
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Figure 11. Graceful degradation of thalamocortical ensembles. A, To 
test further whether SI cortex and VPM ensembles function as a single 
entity, we combined ensembles and measured how sequentially removing 
the best predictor neuron affected performance discriminating four whis- 
kers (B1, B4, El, and E4). Depicted here is the performance of one such 
ensemble; it degraded gracefully. Chance performance was 25%, as 
indicated by the dashed line. B, Here, the graceful degradation of the 
separate SI cortical and VPM ensembles is depicted. Chance perfor- 
mance was 25%, as indicated by the dashed line. 


et al., 1994; Deadwyler et al., 1996; Nicolelis et al., 1997b) and 
artificial neural networks (Nicolelis et al., 1998, 1999). It is im- 
portant to emphasize, however, that the application of statistical 
pattern recognition techniques to “decode” the identity of a 
stimulus does not imply that these methods bear any resemblance 
to the actual mechanisms through which the nervous system 
represents tactile information. Instead, they offer us a way to 
quantify ways in which information could be embedded in neural 
activity patterns. 


Behavioral relevance 

Two important facts must be considered when interpreting our 
results. First, our data were collected from anesthetized rats. 
Second, our analyses were all based on whether neural ensembles 
can discriminate 1 out of x number of whiskers. Both of these 
issues bear on the relevance of our data to the behaving animal. 
After all, during natural exploratory behaviors, rats use their 
whiskers actively and make multiple contacts with objects in their 
environment. 

The dynamic and distributed nature of ensemble responses to 
single whisker deflections in the anesthetized rat is indistinguish- 
able in both its temporal evolution and spatial extent from re- 
sponses seen in the awake animal in certain behavioral states 
(Chapin and Lin, 1984; Fanselow and Nicolelis, 1999). Chapin 
and Lin (1984) found that SI RFs are qualitatively the same in 
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both the anesthetized and awake conditions. If anything, RFs 
were slightly larger in the awake state. This finding is supported 
by more recent studies using more quantitative analyses in which 
it has also been shown that, depending on the behavioral state, 
the system appears optimized for the detection of simple stimuli 
(Fanselow and Nicolelis, 1999). Thus, we believe that our sensory 
responses in the anesthetized condition are an accurate represen- 
tation of the responses one would see in at least some behavioral 
states in the awake animal. 

Our approach in this study was to determine the extent to 
which ensembles could distinguish between the deflection of one 
whisker among others. Because the rat uses the entire caudal 
mystacial pad simultaneously to actively detect the spatial at- 
tributes of its environment (Carvell and Simons, 1990), we asked 
whether our ensembles (or single neurons) could distinguish 
between one whisker out of 4-16 other whiskers. Within the 
context of natural whisking behavior, such a task is quite reason- 
able. Brecht et al. (1997) have elegantly shown that the caudal 
whiskers, unlike the finer-grained rostral whiskers, are primarily 
used to detect the spatial location of objects or openings. Based 
on morphological and behavioral considerations, these authors 
conclude that the major information format in the barrel cortex is 
the “binary touched/untouched signal.” Furthermore, it has been 
shown repeatedly that animals can accurately perform spatial 
tasks with only one whisker (Hutson and Masterton, 1986; Harris 
et al., 1999), showing again that a single whisker is an appropriate 
“sensory unit” to investigate. Within this framework, our use of 
the 1 out of x whiskers paradigm for ensemble performance is 
quite reasonable. The next experimental step ought to involve 
more complex multi-whisker discriminations in behaving animals. 


Single neurons versus neural ensembles 

Our study indicated that ensembles of SI cortical and VPM 
neurons were several times better than single neurons at identi- 
fying the location of a stimulus on the whisker pad on a single 
trial basis. Moreover, ensemble performance degraded gracefully 
when neurons were removed, one by one, from the population. 
These results, combined with the fact that single whiskers can 
elicit a spatiotemporally complex response from a large portion of 
SI cortex (Armstrong-James et al., 1992; Kleinfeld and Delaney, 
1996; Masino and Frostig, 1996; Moore and Nelson, 1998; Peter- 
son et al., 1998; Ghazanfar and Nicolelis, 1999) and VPM 
(Nicolelis and Chapin, 1994), strongly suggest that despite their 
modular anatomy, the rat SI cortex and VPM neurons may rely on 
distributed encoding strategies to identify stimulus location on a 
single trial basis. 

Along similar lines, in the auditory cortex of the cat, Middle- 
brooks et al. (1994, 1998) found that single neurons were also 
broadly tuned for sound location and suggested that a distributed 
population code would be required to encode the location of 
auditory stimuli at the accuracy levels observed psychophysically. 
Thus, because the activity of a broadly tuned neuron is unable by 
itself to provide unequivocal information concerning stimulus lo- 
cation, such activity is only meaningful in the context of the 
activity of other neurons (Erickson, 1968). Importantly, there is 
not a topographic map of auditory space in the cortical area where 
these neurons were sampled from, suggesting that such maps are 
not necessary for coding spatial location with great precision. 

Our data suggest that single neurons or even small, localized 
groups of neurons are, by themselves, inefficient processors of 
sensory information. It has been argued, however, that individual 
neurons may be the dominant coding units for near-threshold 
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stimuli. This “lower envelope principle” states that sensory 
thresholds are set by the sensory neurons that have the lowest 
threshold for the stimulus used (Barlow, 1995). Several points 
argue against the possibility of single neuron decoders in the rat 
somatosensory system. First, receptive fields mapped with smaller 
whisker deflections (i.e., low-threshold stimuli) than used in this 
study and previous studies (Nicolelis and Chapin, 1994) have 
comparably sized receptive fields (Armstrong-James and Fox, 
1987; Simons and Carvell, 1989). Thus, the spatial tuning of 
neurons is similar even for smaller whisker deflections than those 
used in the present study. Second, an optical imaging study, in 
which whisker deflection amplitudes were parametrically manip- 
ulated, demonstrated that even weak stimuli elicit a response that 
extends well beyond a single barrel cortical column in SI cortex 
(Peterson et al., 1998). To reiterate, despite the anatomical mod- 
ularity and topography in both SI and VPM neurons, stimulation 
of each whisker activates cells in locations beyond its isomorphic 
barrel or barreloid. Within a barrel column, for example, individ- 
ual neurons receive synaptic inputs from multiple whiskers 
(Moore and Nelson, 1998; Zhu and Connors, 1999). Such inter- 
connectivity necessarily gives rise to large, distributed responses. 

For the barrel cortex, layer V neurons have, on average, the 
largest RFs (Simons, 1978; Chapin, 1986). It is therefore possible 
that single neurons or smaller groups of neurons in other barrel 
cortical layers could accurately distinguish the location of tactile 
stimuli. For example, although layer [TV neurons have large and 
dynamic multi-whisker RFs at the subthreshold level (Moore and 
Nelson, 1998), these are ultimately reduced in size by inhibitory 
interactions resulting in suprathreshold RFs of only one or two 
whiskers. Nevertheless, the majority of supragranular and infra- 
granular layer neurons of barrel cortex and VPM neurons have 
large, multi-whisker RFs (Simons, 1978; Chapin, 1986; Nicolelis 
and Chapin, 1994), and it is the infragranular neurons that are the 
primary source of SI output. Indeed, neurons in layer V send 
axons to various intracortical and subcortical targets (Killackey et 
al., 1989; Koralek et al., 1990). Thus, the study of layer V ensem- 
bles gives a more accurate perspective of the neural activity that 
downstream targets have to decode. 


VPM neural ensembles perform better than SI cortex 

Both single neurons and ensembles of the VPM nucleus per- 
formed better than SI cortex for 1 out of 16 whisker discrimina- 
tions. This is interesting in light of the fact that VPM neurons 
have larger receptive fields than SI cortex; Le., the tuning of 
neurons becomes sharper from thalamus to cortex (Simons, 1978; 
Armstrong-James and Fox, 1987; Simons and Carvell, 1989; 
Nicolelis and Chapin, 1994; Ghazanfar and Nicolelis, 1999). Rate 
of decay data also suggested that fewer VPM neurons are needed 
to encode stimulus location when compared with SI layer V 
neurons. Perhaps this difference in performance can be attributed 
to the fact that VPM neurons fire at a higher rate under anes- 
thetized (Simons and Carvell, 1989; Nicolelis and Chapin, 1994; 
Ghazanfar and Nicolelis, 1999) and awake (Nicolelis et al., 1995) 
conditions and exhibit greater temporal modulation of their re- 
sponses than SI cortical neurons (Fig. 94). Thus, under these 
conditions VPM ensembles provide information in more dimen- 
sions for encoding stimulus attributes. In the auditory system, it 
was found that sharper tuning of neurons in later stages of the 
pathway resulted in more efficient population vector coding of 
sound localization than earlier stages with more broadly tuned 
neurons (Fitzpatrick et al., 1997). It is conceivable that this 
difference between our results and those of Fitzpatrick et al. (1997) 
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for the auditory system is attributable to the fact that our ensemble 
analysis incorporated temporal information across neurons. 

Although we did not test the role of this property in the current 
study, “bursting” may also play a role in coding stimulus location. 
Bursts are characterized as a series of spikes with short interspike 
intervals, and neurons in both layer V (Amitai and Connors, 
1995) and thalamus (Godwin et al., 1996) may display bursting 
behavior. The relevance of bursts for information encoding and 
transmission in the somatosensory thalamocortical pathway 
awaits further study. 


Inseparability of rate and temporal coding parameters 
Several single neuron studies have demonstrated that the tempo- 
ral modulation of firing rate can carry a significant amount of 
stimulus-related information (Richmond et al., 1990; McClurkin 
et al., 1991a; Middlebrooks et al., 1994, 1998; Victor and Purpura, 
1996). In these studies, single neurons were shown to encode 
various features (color, spatial frequency, etc.) simultaneously 
when time was used as a coding dimension. These results support 
the idea that different features of a stimulus do not need to be 
encoded by distinct populations of neurons each devoted to a 
particular stimulus. Instead, the same population of neurons 
could encode multiple stimulus attributes simultaneously by using 
distinct encoding strategies (e.g., firing rate, time-modulation of 
rate, correlated activity, etc.) to represent each of these features 
(Nicolelis et al., 1998). 

The importance of the temporal dimension in our data was 
demonstrated by showing that decreasing the temporal resolution 
of neural ensemble response resulted in significant decreases in 
performance for both SI cortex and VPM ensembles. Our data 
suggest, therefore, that both the number of spikes and the tem- 
poral modulation of the ensemble firing can carry information 
regarding stimulus location. Firing rate differences among neu- 
rons in SI cortex and VPM ensembles, however, were still suffi- 
cient to encode stimulus location several times above chance 
levels. Similar results have been reported for area SII in primates 
(Nicolelis et al., 1998). Thus, the ensemble coding of tactile 
stimulus location seems to be best represented when both the 
temporal modulation of the neural ensemble response and the 
average firing rate are taken into account. 

Other forms of temporal coding—the phase relationships and 
potentially other forms of correlated activity—were tested by 
spike shifting, a procedure that randomly jitters individual spike 
trains relative to one another, and by trial shuffling, a procedure 
aimed at disrupting the intratrial covariance structure of ensem- 
ble responses. Spike shifting, but not trial shuffling, resulted in 
significantly degraded ensemble performance for both SI cortex 
and VPM ensembles. This suggests that phase relationships be- 
tween the stimulus-locked modulation of firing rate changes 
across the ensemble encode stimulus location in the thalamocor- 
tical loop. This type of covariance structure among neurons is 
maintained in trial shuffling but disrupted in spike shifting. 

This study also revealed that the role of correlated activity 
between neurons in ensemble performance increased as a func- 
tion of the difficulty of the discrimination: as the number of 
whiskers to discriminate among increased, so did the contribution 
of correlated activity as a coding domain. Related to this, the 
number of neurons needed to encode stimulus location did not 
increase linearly as the number of whiskers increased but instead 
reached plateau between 12 and 16 whiskers (Fig. 4B). We 
speculate that these results may be interpreted as an indication 
that more neurons are not necessarily needed because of a cor- 
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responding shift toward the increased use of different coding 
dimensions. This gives rise to the hypothesis that the encoding 
mechanism selected by the neural ensemble may be task depen- 
dent. Thus, under different circumstances, such as behavioral 
states (Fanselow and Nicolelis, 1999), the same neural ensemble 
may take advantage of distinct strategies according to the context 
in which a particular computation is performed. 

It has been argued that covariation of neural activity and the 
temporal discharge patterns of cortical neurons transmit little or 
no information and that rapid changes in firing rate are the sole 
information channel for coding (Shadlen and Newsome, 1998). 
Our data and previous studies (Richmond et al., 1990; McClurkin 
et al., 1991a; Middlebrooks et al., 1994, 1998; Victor and Purpura, 
1996; Dan et al., 1998; Nicolelis et al., 1998) argue for a more 
balanced account of the role of time in neural coding in the 
thalamocortical loop. Indeed, our results demonstrate that de- 
creasing the temporal resolution of the ensemble response (but 
keeping spike count information unchanged) and disrupting the 
covariance structure among cortical and thalamic neurons can 
significantly degrade the discrimination performance of thalamo- 
cortical ensembles. Nevertheless, ensemble performance subse- 
quent to both of these manipulations always remained above 
chance, suggesting that firing rate changes do play a major role in 
the transmission of sensory information. 


The primary somatosensory cortex and thalamus 
function as a single unit 

Several findings in this study argue in favor of the view that in the 
rat somatosensory system, SI cortex and VPM neurons function 
as a single entity in the discrimination of stimulus location on a 
single trial basis. First, measurements of the raw ensemble re- 
sponses of simultaneously recorded cortical and thalamic ensem- 
bles revealed concurrent activity for several milliseconds of post- 
stimulus time (Ghazanfar and Nicolelis, 1997). Second, the 
temporal analysis of SI cortical and VPM ensemble performance 
revealed that above chance performance occurred in both struc- 
tures concurrently. Third, dropping neurons one at a time from 
thalamocortical ensembles resulted in the graceful degradation of 
performance. Finally, the two structures showed some redun- 
dancy of information in the coding of stimulus location. Coupled 
with the extensive data on the reciprocal anatomical connections 
between these two structures (Chmielowska et al., 1989; Bourassa 
et al., 1995), these results suggest the existence of tightly related 
functional neural ensembles that could be used, among other 
things, to encode tactile stimulus location. In support of this con- 
tention, neurons in the primate somatosensory and visual system 
also seem to encode information concurrently in reciprocally con- 
nected structures (McClurkin et al., 1991b; Nicolelis et al., 1998). 


Conclusions 

One of the potential benefits of topographic maps in the sensory 
systems of vertebrates is the ability to easily identify the location 
of a stimulus: localized groups of neurons respond specifically to 
the presence of stimulus in a restricted portion of the sensory 
space. Yet despite the precise topographic arrangement of mod- 
ules along the rat trigeminal somatosensory pathway, this sensory 
system does not appear to restrict its encoding repertoire to a 
local coding scheme. Instead, our data demonstrate that a distrib- 
uted coding scheme, in which the participation of a large number 
of neurons located in many different modules across the thalamo- 
cortical pathway is necessary, may be used by this system to 
compute the location of a tactile stimulus on a single trial basis. 
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Within this scheme, both the spatial and temporal characteristics 
of neural ensemble firing convey information. Moreover, our data 
suggest that the strategies that the system may use change as a 
function of the number of stimulus locations to discriminate, a 
measure of the degree of difficulty. 

In summary, the representation of sensory features appears to 
arise from the dynamic interactions among neurons within and 
between brain structures, which include various coding strategies. 
We also propose that, depending on behavioral states and/or the 
task at hand (or whiskers, in this context), the CNS may rely on 
different strategies to solve the same problem. In this framework, 
pure firing rate coding and multiple time codes may coexist in the 
same ensemble. 


APPENDIX 


The LVQ classification system used for this study is a multilayer, 
feedforward ANN with full connectivity. The first layer is simply 
the input or pattern vector. The second layer contains an ANU 
for each pattern to be discriminated. Each ANU has a weight 
vector with an equivalent number of elements as the input (pat- 
tern) vector. There is no bias value for these AN Us. The output 
value of each ANU is determined by the following Euclidean 
distance function: 


a(W;,1) = —|II — Wil, (1) 


where a; is the output value for the i‘* ANU, W, is the weight 
vector, and I is the input vector for the system. The unit with the 
greatest output value has the weight vector that is closest in 
Euclidean distance to the input vector. This is known as the 
“winning” ANU. 


The LVQ learning rule 

Each ANU is responsible for recognizing input patterns. Because 
the class of patterns that each of these ANUs must find is 
predetermined (supervised learning), it must be penalized for 
finding a pattern of the wrong class and rewarded for finding a 
pattern of the correct class. This is realized by the optimized 
LVQ (Kohonen 1997). 

Let c define the index of the winner in the second layer: 


c = arg min{||I — Will} 


and let a;(t) the learning-rate factor assigned to each W,, then: 
W(t + 1) = [1 — s(t) a(t) W(t) + s(t) a(f)I(t), (3) 
where 
s(t) = +1 if Lis classified correctly 


s(t) = —1 if Lis classified incorrectly, 


and after each learning step: 


a(t) = [1 — s(t) a(t)] a(t — 1). (4) 


This classification system searches for patterns closest in Euclid- 
ean distance to one of our input weight vectors, so each of these 
weight vectors is a codebook vector that the system will use for 
classification. The LVQ algorithm shown moves these codebook 
vectors closer to properly classified input training vectors and 
away from improperly classified training vectors by the learning 
rate a(t), which decreases after each learning step. 


Ghazanfar et al. e Coding by Somatosensory Thalamocortical Ensembles 


Initialization 

The weight vectors of the ANUs are all initialized to the same 
value: the midpoint of the range of values of the input vectors. 
They differ, however, in what class of pattern they are assigned to 
recognize. Because this is a supervised learning algorithm, each 
ANU is preassigned to one class during initialization. During 
training, these AN Us learn to recognize the class to which they 
are preassigned. In general, there must be at least as many AN Us 
as there are classes to be discriminated, although it is possible to 
have more ANUs than classes. In this study, two ANUs are 
assigned to each class so that there are twice as many ANUs as 
there are classes. 


Training 

Training the LVQ network involves executing the LVQ learning 
tule for all the input vectors repeatedly until the ANU weight 
vectors have moved as close as possible to their assigned class of 
input patterns. In this study we chose a sufficiently large number 
of iterations to ensure that this occurs: two times the product of 
the number of input vector patterns and the number of ANUs. 
This number of training iterations was determined empirically to 
be sufficient such that the weights of the AN Us have become the 
optimal codebook vectors for classification of input patterns. 


Testing 

To test the LVQ network we present it with trials that were not 
used during training. One ANU weight vector will be the “win- 
ning” ANU, and the class to which it was preassigned is the 
predicted classification of the input pattern. If this matches the 
actual class of the input pattern, proper classification has been 
achieved, and if it does not then an improper classification has 
occurred. 
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increasingly predicts behaviour 
outcomes during learning 

of a motor task 
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When an animal learns to make movements in response to 
different stimuli, changes in activity in the motor cortex seem 
to accompany and underlie this learning’~°. The precise nature of 
modifications in cortical motor areas during the initial stages of 
motor learning, however, is largely unknown. Here we address 
this issue by chronically recording from neuronal ensembles 
located in the rat motor cortex, throughout the period required 
for rats to learn a reaction-time task. Motor learning was demon- 
strated by a decrease in the variance of the rats’ reaction times and 
an increase in the time the animals were able to wait for a trigger 
stimulus. These behavioural changes were correlated with a 
significant increase in our ability to predict the correct or 
incorrect outcome of single trials based on three measures of 
neuronal ensemble activity: average firing rate, temporal patterns 
of firing, and correlated firing. This increase in prediction indi- 
cates that an association between sensory cues and movement 
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emerged in the motor cortex as the task was learned. Such 
modifications in cortical ensemble activity may be critical for 
the initial learning of motor tasks. 

Traditional theories of cortical function propose that information 
in the central nervous system is represented primarily by the firing 
rate of single neurons”. Learning new motor contingencies should 
therefore primarily involve changes in the average firing of neurons 
located in sensorimotor cortical areas. Behaviourally relevant 
information can also be represented in temporal patterns of 
neuronal firing (on a ms scale) and correlated neuronal activity’. 
It is not known whether these other schemes of neuronal represen- 
tation are significant in the physiological modifications associated 
with the initial learning of a motor task. To address this issue, we 
compared the potential contribution of three neuronal coding 
schemes (average firing rate, temporal patterns of firing, and 
correlated activity) for predicting the behavioural outcome of 
single trials throughout the period required for rats to learn a 
reaction-time task. We also compared concurrent neural ensemble 
and electromyography (EMG) recordings to establish that any 
increase in single trial prediction by ensemble activity was not 
due to differences in movements on the correct and error trials. 

Rats were trained to perform a reaction-time task in which they 
were required to hold down a response lever throughout a variable 
interval (the foreperiod, 400-800 ms) and to release the lever 
in response to vibrotactile or auditory trigger stimuli with a short 
(< 1s) reaction time (Fig. la). Correct trials occurred when the rats 
sustained the lever press until the presentation of the trigger stimuli 
and released the lever with a short reaction time. Error trials 
occurred when the rats released the lever before the trigger stimulus. 
During the first of week of training, the rats exhibited significant 
increases in the time they were able to wait for the trigger 
stimulus, from 422.3 +30.9ms (day 1) to 577.8 + 24.8ms (day 
7; repeated-measures analysis of variants (ANOVA): P < 10°3 
Fig. 1b). Although the rats’ reaction times during these sessions 
did not change (day 1: 320.8 + 41.5ms; day 7: 271.6 + 32.9 ms; 
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Figure 1 Improvements in behavioural performance parallelled increases in the neuronal 
predictions of trial outcomes. a—c, As the rats learned the task (a), they sustained lever 
presses over longer foreperiods (b) and exhibited decreased variance in their reaction 
times (c). d, Predictions of trial outcomes based on neuronal activity increased with 
training and, by day 6-7, achieved levels equal to those in fully trained rats (upper lines, 
mean + s.e.m). Chance discrimination was 50% (lower dashed line). Temporal patterns 
of firing (from smoothing over 10-ms intervals) predicted trial outcomes better than 
average firing rates (combined over 200 ms). Likewise, predictions based on correlated 
firing were significantly better than those based on decorrelated ensemble activity. 
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repeated-measures ANOVA: P > 0.05), the variance of the rats’ 
reaction times decreased significantly (day 1: 165 + 24.2 ms; day 7: 
44.5 + 6.6 ms; repeated-measures ANOVA: P < 0.02; Fig. 1c). After 
these training sessions, our animals performed the task as well as 
animals in previous studies of reaction-time behaviour in rats, that 
is, better than 70% correct trials!®. 

Chronic and simultaneous recordings were obtained from 
neurons (25.4 + 1.3 neurons per session, mean + s.e.m., range 14 
to 32 neurons, total 711) in the forelimb representation of two 
motor areas of the cerebral cortex (medial and lateral agranular 
cortical areas, mAG and IAG) of six rats. To study the effect of 
learning on neuronal activity, spike trains in the 200-ms interval 
before lever release were compared for correct and error trials. The 
capability of three different types of neuronal coding schemes to 
predict the outcomes of single trials as correct or error was 
quantified using artificial neural network (ANN) methods'”". 
The ANNs were trained with neuronal ensemble responses from 
subsets of trials and tested with the remaining trials (Fig. 2) for each 
training session throughout the learning period. The degree to 
which confusion matrices from the ANNs differed from chance was 
then evaluated using information theory” and was expressed as bits 
of information. 

The majority of cortical neurons (477, 67.1%) were active when 
the rats pressed and released the lever (Fig. 2). However, over the 
seven training sessions, the responses of only nine of these neurons 
provided more than 0.1 bits of information about trial outcome. 
That is, 98.1% of the single neurons alone did not predict trial 
outcome accurately (overall average of 59.3 + 0.3% correct, 0.015 
0.001 bits for all training sessions). The average amount of informa- 
tion represented by the single neurons was 0.011 + 0.001 bits on the 
first day of training and 0.024 + 0.004 bits on the seventh day of 
training. Permutation tests” showed that these information values 
are not statistically significant and can be generated by chance alone. 


Correct Error 


ni7 


n20 


Figure 2 Neuronal ensemble responses associated with correct and error trials. a, Single 
neurons (n) in the lateral (IAG) and medial (mAG) agranular areas of motor cortex fired 

differently on correct and error trials (bin, 10 ms). The bars below each plot show the 200- 
ms period analysed in this study. b, Different temporal modulations of firing on correct and 
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By contrast, predictions of trial outcome based on the average 
firing rate (using a single 200-ms bin) of the same cortical neurons 
analysed together as ensembles improved significantly in the first 
seven days of training, from 55.3 + 2.5% (0.013 + 0.004 bits, day 1) 
to 70.3 + 1.4% (0.073 + 0.017 bits, day 7, P < 0.01, repeated 
measures ANOVA, Fig. 1d). While the amount of information 
represented by average ensemble firing rate on the seventh day of 
training was small, permutation tests showed that it was highly 
significant (P ~ 0.30, day 1; P = 0.02, day 7). This improvement 
in trial prediction over learning occurred in the absence of any 
significant change in overall firing rates for the neuronal ensembles 
(20.6 + 1.6 Hz for all sessions, Kruskal—Wallis test: P = 0.573). 

Next, we observed that single trial classification improved sig- 
nificantly when neuronal spike trains were smoothed over 10-ms 
intervals (Fig. 3a), from 66.2 + 2.9% (0.087 + 0.020 bits, day 1) to 
79.1 + 4.2% of the trials correct (0.227 + 0.060 bits, day 7). This 
corresponded to more than a 200% increase in trial prediction based 
on information theory (P < 0.01, repeated measures ANOVA, 
Fig. 1d). Predictions based on fine temporal patterns of firing 
(10-ms intervals) on the seventh day of training yielded three 
times more information than average firing rates (200-ms interval) 
and were better than those based on average firing rates for all days 
of training (ANOVA: P < 10“). In fully trained animals (1 = 6), the 
outcomes of single trials were predicted equally well if spikes were 
smoothed over 5-ms intervals (79.6 + 1.9% correct, 0.207 + 
0.032 bits) or 10 ms (77.3 + 1.4%, 0.178 + 0.025 bits). However, 
predictions were significantly worse (ANOVA: P < 0.005) when 
spikes were smoothed over 20-ms intervals (71.8 + 0.7% correct, 
0.105 + 0.008 bits) or combined over 200 ms (63.6 + 1.9% correct, 
0.038 + 0.011 bits). These results indicate that fine temporal 
integration (10 ms or less) of firing across an ensemble of cortical 
neurons may encode the contingency between trigger stimuli and 
movement execution. 


b Release lever: Correct 


IAG 


mAG 


IAG 


mAG 


Time (s) 


Per cent maximum response 
a 
-100 0 100 


error trials occurred over the neuronal ensembles around lever release (time = 0). Each 
horizontal line represents the firing of a neuron over time (in seconds). Neuronal firing was 
normalized to be —100 (purple) to +100% (red) of the maximum firing rate in the 
ensemble (bin, 10 ms; responses smoothed with a 5-ms moving average filter). 
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We also measured how well correlated firing predicted single trial 
outcomes over the course of learning. Independent component 
analysis (ICA; refs 21, 22) was used to quantify zero-lag correlations 
between groups of neurons (Fig. 3b). Prediction of single trial 
outcome based on the independent components improved signifi- 
cantly over learning, from 60.9 + 4.0% (0.049 + 0.022 bits on day 1) 
to 73.1 = 2.4% (0.167 + 0.044bits on day 7, repeated measures 
ANOVA: P < 0.01, Fig. le), and provided more than twice the 
information obtained from average firing rate. The amount of 
covariance accounted for by the independent components did not 
change over the training period (day 1: 42.6 + 2.3%; day 7: 45.0 + 
4.5%; repeated measures ANOVA: P = 0.85). This result indicates 
that although the degree of correlated activity within the ensembles 
did not change, the information about trial outcome contained in 
the correlated activity increased with learning. The distributions of 
the independent components over the neuronal ensembles revealed 
patterns of correlated firing by subsets of the neurons that were not 
equally distributed across the motor cortex (Fig. 3b). 

To demonstrate further that the information accounted for by 
ICA was due to correlated firing, a ‘spike-shifting’ procedure was 
used to reduce correlations across the neuronal ensemble without 
altering the firing rates of the individual neurons. This procedure 
resulted in a significantly lower percentage of correctly classified 
trials by the ANNs throughout learning (n = 4, P < 0.001, Fig. le). 
These values of information were not significantly different from 
those obtained with the average firing rates of the same ensembles 
(Student’s t-test: P > 0.05). Thus, correlated firing, as detected by 
ICA, proved to be a potential means of encoding information in 
addition to that available in average neuronal ensemble firing rates. 

One possible confound for this study was that the occurrence of 
sensory stimuli on correct but not error trials could have been the 
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basis of the discriminations. However, several results discount this 
possibility. First, very few neurons exhibited any response to the 
vibrotactile and none to the auditory stimuli. Second, in the rare 
occasions when a tactile response was observed, the latency and 
duration of the responses were brief. As our data analysis interval 
was 200 ms before movement onset, and the average reaction time 
was 271.6 + 32.9 ms (day 7), these short-lasting tactile responses did 
not have a major impact on our analyses. Also, the time interval 
accounting for most of the single trial prediction (Figs 2 and 3) 
occurred 100 ms before movement onset. Thus, it seems unlikely 
that sensory responses of the cortical neurons accounted for the 
improvement in trial prediction observed over learning. 

Another potential confound is that the cortical neuronal patterns 
that produce predictions of single trials may have reflected differ- 
ences in motor output between correct and error trials. Several types 
of control data imply that this is not probable. First, video record- 
ings of the animals showed no difference in overt behaviour between 
the types of trials. Second, there was no statistical difference in lever 
trajectories between correct and error trials. Third, EMG activity 
(Fig. 4a, b), recorded from several of the main muscles involved in 
lever release in two control animals, provided insignificant amounts 
of information about whether a single trial was correct or not (rat 1, 
2 muscles: 62.2 + 2.0% correct, 0.008 + 0.003 bits; rat 2, four 
muscles: 61.8 + 1.1% correct, 0.029 + 0.007 bits; P > 0.1; Fig. 4c). 
Fourth, in both control animals, single trial classification achieved 
using groups of muscles was not better than that obtained using the 
best individual muscle in each animal (that is, there was no additive 
effect across muscles). Finally, when neuronal ensemble activity in 
motor cortex was recorded simultaneously with multiple EMGs in 
the same sessions, we observed that single trial discrimination based 
on neuronal data continued to be highly significant even though 
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Figure 3 Predictions of trial outcome were based on temporal patterns of firing and the final 200 ms before movement (data from two fully trained subjects). Neurons that 


correlated activity. a, Temporal patterns of neuronal firing (about 5-10 ms) predicted 
correct (black) and error (red) trials better than average firing rates (box plots: box is 
interquartile range (IQR), line is median, and dots are extreme values, that is, greater than 
1.5 times the IQR), which were not different on correct and error trials. b,e, Independent 
component analysis (ICA) detected correlations between subgroups of neurons (b) during 
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fired together have the same colour boxes, and the size of the boxes reflects the 
correlation between the neurons. Trial outcomes were predicted significantly above 
chance due to the occurrence of different patterns of correlated firing (c) on correct (black) 
and error (red) trials. 
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Figure 4 Muscle activations and response kinematics were similar on correct and error 
trials. a, Correct trial. b, Error trial. The triceps brachii (black) showed increased EMG 
activity during lever pressing, while the trapezius (red) showed increased activity as the 
limb was placed on the lever and during lever release (vertical lines). Behavioural events: 
1, placing paw on lever; 2, pressing lever; 3, releasing lever. ¢, Average EMG activations 
and lever kinematics were equivalent on correct (black) and error (red) trials. d, Neuronal 
ensemble activity recorded in one of these rats provided far more information (asterisk, 
P <0.01) than the multiple EMG or lever position signals, which did not predict trial 
outcome better than expected by chance (P > 0.1). 


EMGs never achieved significance (Fig. 4c). In summary, these 
control experiments indicate that the improvement in predictions 
of trial outcomes over the learning period were not because of 
differences in the movements between correct and error trials. 

Our results demonstrate that the ability of neuronal ensembles in 
the motor cortex to encode sensorimotor contingencies increases 
during motor learning. We found that the average firing rate, 
temporal structure and correlated activity of neuronal ensembles 
in the motor cortex changes during learning to represent new 
behaviourally relevant information. As reproducible results were 
obtained in four rats by randomly sampling 20-30 neurons per 
animal, these changes probably involve widespread territories of the 
rat motor cortex. Based on these observations, we propose that new 
functional associations between subgroups of cortical neurons may 
emerge as animals learn the new sensorimotor contingencies that 
are required to solve motor tasks. 


Methods 


For more detailed methods see Supplementary Information. 


Behavioural task 


We recorded neuronal responses during learning from four adult, male rats. Neuronal 
responses were recorded from two additional animals only after they were fully trained, 
and two more animals were used for control studies using EMG. The rats were moderately 
deprived of water and trained to press a lever and release it in response to a trigger stimulus 
for a liquid reward. The foreperiod was initially 250 ms and was incremented by 50— 
100 ms whenever the rats made three consecutive correct trials. Once the foreperiod was 
greater than 600 ms and the animal performed the task consistently, the foreperiod varied 
randomly between 400-800 ms. All events during training and recording were controlled 
by computer. Behaviour of all rats was monitored by a video system, and by a position 
sensor attached to the lever in the two rats used for EMG controls. 


Electrophysiological recordings 


All rats were treated in accordance with NIH guidelines. Procedures for chronically 
implanting arrays of microwires (NB Labs, Dennison, Texas) and data acquisition, using a 
Many Neuron Acquisition Program (Plexon, Dallas, Texas), are described elsewhere”. 
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After the completion of all experiments, select recording sites were marked with 
electrolytic lesions before formalin perfusion, and the locations of microwires were 
reconstructed using histological techniques. 

Multiple bipolar patch electrodes (MicroProbe, Potomac, Maryland) were implanted 
subcutaneously into two rats to record EMG ensemble activity chronically™. In the first 
rat, we recorded the activity of the triceps brachii (action: elbow extension) and the 
thoracic portion of trapezius (rotation of the limb at the shoulder). In the second rat, we 
recorded EMG activity from the anterior portion of deltoideus and pectoralis in addition 
to trapezius and triceps. EMG signals were amplified (SA Instrumentation, Encinitas, 
California), filtered at 20—500 Hz and sampled at 1,000 Hz. 


Data analysis 


Our strategy for statistical analysis of neuronal ensemble data sets is described 
elsewhere'*”>”®, Single trial classification based on average neuronal firing rates was 
assessed by first summing all spikes for each trial over a 200-ms interval before lever 
release. These spike counts were then fed into an artificial neural network (ANN) that used 
the learning vector quantization (LVQ) algorithm”. Training and testing of artificial 
neural networks were performed on each behavioural training session in each subject to 
obtain the numbers depicting single trial classification. Leave-one-out cross-validation” 
was used to estimate error rates and the confusion matrices for each data set. The degree to 
which confusion matrices from the ANNs differed significantly from chance was then 
evaluated using information theory’ and permutation tests”, and was expressed as bits of 
information. 

Peri-event histograms (1-ms bins) were constructed for the correct and error trials for 
the 200-ms interval before lever release. The spike trains were then smoothed over 5-, 10- 
or 20-ms intervals by low-pass filtering and re-sampling (Fig. 3a). Firing patterns 
associated with correct and error trials were identified for each neuron using a wavelet- 
based method”, which essentially identified temporally specific, salient features in the data 
and thereby reduced the number of variables used to train the ANNs. Classification using 
the wavelet features was done as described above. 

Independent component analysis (ICA) was used to assess how well correlated activity 
among different groups of cortical neurons predicted single trial outcome”. First, 
principal component analysis” was used to reduce the dimensionality of data from 200-ms 
intervals before lever release. An extended independent component algorithm” was then 
used to rotate the weights for the largest principal components to make the functions as 
statistically independent as possible. Scores for the resulting independent components 
were then computed for each 1-ms interval. These scores were then preprocessed with the 
wavelet algorithm and analysed with the same artificial neural network methods as 
described above. To test the notion that the ICA procedure detected correlated firing, the 
spike trains were decorrelated using a spike-shifting procedure described elsewhere”. 
Single trial classification based on digitally rectified EMG activity or lever position data 
was done using the same procedures as for the neuronal spike trains. 
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Two electroneutral, Na‘-driven HCO; transporters, the Na‘- 
driven Cl'/HCO; exchanger and the electroneutral Na‘/ HCO; 
cotransporter, have crucial roles in regulating intracellular pH in 
a variety of cells, including cardiac myocytes’”, vascular smooth- 
muscle**, neurons’ and fibroblasts®; however, it is difficult to 
distinguish their Cl dependence in mammalian cells. Here we 
report the cloning of three variants of an electroneutral Na*/HCO; 
cotransporter, NBCn1, from rat smooth muscle. They are 89-92% 
identical to a human skeletal muscle clone’, 55-57% identical to 
the electrogenic NBCs and 33-43% identical to the anion 
exchangers’. When expressed in Xenopus oocytes, NBCn1-B 
(which encodes 1,218 amino acids) is electroneutral, Na*-dependent 
and HCO;-dependent, but not Cl -dependent. Oocytes injected 
with low levels of NBCn1-B complementary RNA exhibit a Na* 
conductance that 4,4-diisothiocyanatostilbene-2,2'-disulphonate 
stimulates slowly and irreversibly. 

We designed degenerate primers to identify NBC-related com- 
plementary DNAs from rat aorta based on an amino-acid sequence 
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comparison of several NBCs’~"”, a Caenorhabditis elegans expressed 


sequence tag (EST) clone (GenBank accession number Z75541) and 
rat anion exchangers’’. We cloned a 294-base pair (bp) polymerase 
chain reaction (PCR) product, the deduced amino-acid sequence 
of which is 76% identical to electrogenic renal and pancreatic 
NBCs. We obtained the 5’ end of the open reading frame by 5’ 
rapid amplification of cDNA ends (RACE) of aorta RNA, and 
obtained the 3’ end by performing PCR on a pulmonary-artery 
cDNA library. 

To test whether our three cDNA fragments represent a single 
transcript, we performed PCR with reverse transcription (RT—PCR) 
using total RNA from rat aorta and primers designed to amplify the 
entire coding region. We obtained three distinct, full-length clones 
(NBCn1-B, C and D). Compared with the NBCn1-D (Fig. 1a), 
NBCn1-B lacks a 36-residue “B’ domain near the carboxy terminus, 
and NBCn1-C lacks a 14-residue ‘A domain near the amino 
terminus (Fig. 1b). The rat NBCn1s are 89-92% identical to a 
human skeletal-muscle cDNA (‘NBC-3/Pushkin’)’, which we pro- 
pose to call NBCn1-A. Figure 1a compares the deduced amino-acid 
sequence of NBCn1-D with the electrogenic rat-kidney NBC (57% 
identity)’ and rat AE2 (33-43% identity)’. Figure 1c shows the 
relationships among selected members of the HCO; transporter 
superfamily. The identity between the NBCnls and electrogenic 
NBCs is a fairly uniform 62% along the membrane-spanning 
regions, but falls off sharply (< 20%) at the amino and carboxy 
termini. The hydropathy plot of NBCn1-D is similar to that of a 
typical electrogenic NBC (Fig. 1d). 

A northern blot of rat tissues (Fig. 2a), using a probe correspond- 
ing to a region common to the NBCn1s, shows substantial hybri- 
dization to a 7.5-kilobase (kb) transcript in spleen and a 4.1-kb 
transcript in testis. The signals are moderate in several other tissues, 
but undetectable in skeletal muscle. Using primers based on con- 
served NBCn1 sequences, we obtained a PCR product of the 
predicted size from spleen, aorta and brain (Fig. 2b). 

To test the functional properties, we injected NBCn1-B cRNA 
into oocytes, and used microelectrodes to monitor membrane 
potential (V,,) and intracellular pH (pH;). Oocytes expressing 
NBCn1-B differ in two principal ways from controls. First, the 
resting V,, was -30 +1 mV (n = 32), ~30mV more positive 
(P < 0.001) than in control oocytes (-60 +3 mV, n = 9). 
Second, the average initial pH; was 7.46 + 0.01 (n = 29), 0.12 units 
higher (P < 0.01) than in controls (7.34 £0.03, n = 5). Thus, 
NBCn1-B mediates net HCO3 influx with the trace amounts of 
HCO; present in the incubation medium. 

In control oocytes injected with water (Fig. 3a), applying 1.5% 
CO,/10mM HCO; caused a rapid pH; fall, followed by a long 
period of stability. In oocytes injected with NBCn1-B cRNA 
(Fig. 3b), adding CO,/HCO; to the bath caused the usual pH; fall, 
followed by a rise owing to HCO; uptake. More importantly, CO./ 
HCO; did not affect V,,. In oocytes expressing an electrogenic NBC, 
CO,/HCO; elicits a rapid hyperpolarization, which may be as large 
as 80 mV (refs 9,10,12). Thus, NBCn1-B mediates electroneutral 
Na*/HCO; cotransport (that is, Na*:HCO3 stoichiometry of 1:1). 
Removing Na* from the bath converted the pH; recovery to an 
acidification (Fig. 3b), consistent with reversal of electroneutral 
Na*/HCO; cotransport. Note that Na* removal hyperpolarized 
oocytes expressing NBCn1-B by 25 + 1 mV (= 10), much larger 
than the 11 +1 mV (n = 9) in controls (P < 0.001). 

When we acidified oocytes expressing NBCn1-B with butyric acid 
rather than CO) (Fig. 3c), pH; failed to recover in the presence of 
Na‘, and failed to acidify in the absence of Na‘. Thus, transport by 
NBCn1-B requires HCO; in both the inward and outward direc- 
tions. Note also that Na* removal in butyrate caused the same large 
hyperpolarization as in CO,/HCO3. 

To test whether NBCn1-B encodes a Na‘*-driven Cl /HCO; 
exchanger, we removed bath CI in the presence of CO2/HCO3 
(data not shown); however, Cl removal did not speed the pH; 
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Abstract 


Previously we have shown that the kinematic parameters of reaching movements can be extracted from the activity of cortical 
ensembles. Here we used cortical ensemble activity to predict electromyographic (EMG) signals of four arm muscles in New World 
monkeys. The overall shape of the EMG envelope was predicted, as well as trial-to-trial variations in the amplitude and timing of 
bursts of muscle activity. Predictions of EMG patterns exhibited during reaching movements could be obtained not only from primary 
motor cortex, but also from dorsal premotor, primary somatosensory and posterior parietal cortices. These results suggest that these 
areas represent signals correlated to EMGs of arm muscles in a distributed manner, and that the larger the population sampled, the 
more reliable the predictions. We propose that, in the future, recordings from multiple cortical areas and the extraction of muscle 


patterns from these recordings will help to restore limb mobility in paralysed patients. 


Introduction 


In recent years, a number of studies have addressed the subject of the 
brain—-machine interface (BMJ), which holds promise for restoring 
motor control to paralysed individuals (Wessberg et al., 2000; Taylor 
et al., 2002; Carmena et al., 2003). Our work (Wessberg et al., 2000; 
Nicolelis, 2001, 2003; Carmena et al., 2003) has shown that using 
multielectrode recordings from multiple cortical areas, movement 
parameters can be reliably extracted from neuronal ensemble activity 
and used to control a robotic manipulator. Here we explore the 
possibility of extracting patterns of muscle activation as recorded 
using surface electromyographic (EMG) techniques from population 
activity of cortical neurons. This study was motivated by the idea that 
predictions of muscle activity extracted from cortical signals could be 
used to allow paralysed patients to control robotic devices or even to 
regain control of their own limbs using functional electrical stimula- 
tion (Degnan et al., 2002). 

Extracting a muscle-like signal from cortical population activity is 
not unlike mimicking Sherrington’s ‘final common path’ (Sherrington, 
1906). For Sherrington, the final common path was represented by a 
motoneuron that received a number of inputs (e.g. descending inputs 
from supraspinal centers, mono- and polysynaptic inputs from muscle, 
tendon and skin afferents, reciprocal innervation of antagonist 
muscles), which could be excitatory or inhibitory, but with the end 
effect on the muscle fibers solely determined by the activity of the 
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motoneuronal pool. Similarly, we asked if muscle activation patterns 
could be extracted from population activity of cortical neurons 
recorded in monkeys performing reaching movements. Our rationale 
was that, although the activity of individual neurons may not match 
muscle activations exactly and may contain signals not related to 
muscle activity per se, the firing pattern of each neuron may contain 
an EMG-correlated component. Therefore, summation of the activity 
of many neurons may allow extraction of muscle signals in a way 
analogous to the integrative action of a motoneuron. 

Previously, limb kinematics were reconstructed with high accuracy 
from ensemble activity recorded from multiple cortical areas in New 
World monkeys (Wessberg et al., 2000). However, reconstructing 
muscle electrical activity could be a very different problem. Although 
in the primary motor cortex (M1) neuronal activity has been shown to 
be related to muscle force (Evarts, 1968, 1969; Ashe, 1997) and to 
EMG activity (Fetz & Finocchio, 1975; Fetz & Cheney, 1980; Bennett 
& Lemon, 1994; Scott, 1997), and EMG signals during stereotypic 
repetitive movements have been predicted from serially recorded M1 
neurons (Morrow & Miller, 2003), it is known that M1 neurons can 
represent higher-order parameters such as the movements of limb 
endpoints irrespective of muscle activation patterns (Mountcastle 
et al., 1975; Georgopoulos et al., 1982, 1986; Schwartz, 1992; Burnod 
et al., 1999) and motor goals independent of limb movements (Shen & 
Alexander, 1997a; Wise et al., 1998; Kakei et al., 1999). The 
predominance of nonmuscle signals in M1 activity has been 
challenged by theoretical (Mussa-Ivaldi, 1988; Todorov, 2000) and 
experimental studies (Sergio & Kalaska, 1997, 1998) that suggested 
that muscle-related signals may account for many of the neuronal 
firing patterns attributed to higher-order representations. Nonetheless, 
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neuronal modulations that are quite distinct from muscle patterns are 
clearly very prominent in nonprimary motor areas (for a review see 
Wise et al., 1997; Johnson et al., 2001; Tanji, 2001; Andersen & 
Buneo, 2002). For example, premotor cortex neurons can exhibit 
preparatory activity that can precede muscle modulations by several 
seconds (Weinrich & Wise, 1982), can encode spatial variables distinct 
from muscle patterns such as orientation of spatial attention (Lebedev 
& Wise, 2001) or motor goals (Shen & Alexander, 1997b), and may 
use nonmuscle spatial reference frames (Kakei ef al., 2001). The 
presence of nonmuscle signals in cortical activity may present a 
problem for extraction of EMG patterns, especially for the general- 
ization of these predictions to a wide range of movements. 

Despite this potential problem, there is evidence that neurons 
widely distributed across sensorimotor cortex may directly influence 
muscle patterns. Cortical areas other than M1 have prominent 
corticospinal projections (Murray & Coulter, 1981; Dum & Strick, 
1991, 2002), and stimulation of both primary and nonprimary motor 
areas elicits limb movements (Mitz & Wise, 1987; Luppino et al., 
1991; Godschalk et al., 1995; Preuss et al., 1996; Graziano et al., 
2002) that range in their complexity depending on the nature and site 
of stimulation. In addition, frontal and parietal areas are strongly 
interconnected (Jones et al., 1978). For instance, primary somatosen- 
sory cortex (S1) receives inputs from M1, which have been suggested 
to prepare S1 for sensory reafference that would result from an 
upcoming movement (Soso & Fetz, 1980; Chapin & Woodward, 
1982; Lebedev et al., 1994; Fanselow & Nicolelis, 1999). M1 is also 
interconnected with dorsal premotor cortex (PMd) and motor areas in 
posterior parietal cortex (PPC), while PMd and PPC are in tum 
interconnected with each other (Wise et al., 1997). 

Because of their interconnectivity and similar activations in many 
instances, frontal and parietal areas appear to process motor informa- 
tion in a parallel, distributed manner. According to the concept of 
distributed processing (Pribram, 1971; Rumelhart et al., 1986; 
Churchland & Sejnowski, 1989; Bridgeman, 1992; Chapin & 
Nicolelis, 1995; Nicolelis, 1998), different cortical areas act as parts 
of a larger neuronal network rather than autonomous processing 
modules. In this scheme, a parameter is represented by the collective 
activity of large ensembles of neurons rather than the activity of a few 
specialized neurons (e.g. ‘grandmother cells’ in vision). Individual 
neurons in the ensemble may not contain all the information about a 
particular parameter, but the ensemble as a whole does. This is 
supported by evidence that increasing neuronal sample size improves 
the ability to predict motor parameters (Humphrey ef al., 1970; 
Wessberg et al., 2000; Carmena ef al., 2003). Moreover, a single 
neuronal ensemble may represent multiple parameters (Wessberg 
et al., 2000; Carmena et al., 2003; Lebedev et al., 2004). With these 
results in mind, we set out to test the hypothesis that information about 
muscle activity patterns could be obtained from multiple cortical areas. 


Materials and methods 
Behavioural task and data acquisition 


All experiments were conducted in accordance with the guidelines for 
animal care and use established by the National Research Council and 
with the approval of the Institutional Animal Care and Use Committee 
of Duke University and Medical Center. Two New World monkeys, one 
squirrel (Saimiri sciureus, Monkey 1) and one owl monkey (4otus 
trivirgatus, Monkey 2), were trained to perform a reaching and grasping 
task as in Wessberg et al. (2000). Each monkey sat comfortably in a 
primate chair with its right hand resting on a small platform that was 
attached to the chair, waist high, and in front of the body. Two barriers, 


one opaque and one transparent, separated the monkey from the tray, 
where an experimenter placed a piece of food in one of four target 
locations. The target positions were equidistant from each other, and 
formed an arc centered at the resting hand position with a radius of 
10 cm and angular span of 90°. The target location for each trial was 
selected in a pseudorandom fashion. Once the food was placed on the 
tray, the opaque barrier was lifted, allowing the monkey to see the food. 
Following a variable delay of 1—5 s, the experimenter raised the 
transparent barrier, after which the monkey reached out, grasped the 
food and brought it to its mouth. There were 160 trials per session, and 
five sessions for Monkey 1 and four for Monkey 2 in which EMG 
signals for both biceps and triceps of the right arm were recorded 
simultaneously with neuronal activity. Additionally, we recorded from 
the extensor digitorum (a wrist extensor) during three sessions in 
Monkey 1, and the deltoid during two sessions in Monkey 2. 

Arrays of 16 or 32 50-11m-diameter stainless steel microwires were 
chronically implanted (Kralik et al., 2001; Nicolelis et a/., 2003) under 
isoflurane anesthesia in the hand/arm regions of PMd, M1 and area 3b 
of S1, and in the dorsolateral aspect of the superior parietal lobule of 
PPC in each monkey. During surgery, heart rate, oxygen saturation, 
blood pressure, respiratory rate, end tidal carbon dioxide, temperature, 
blood glucose and urine output were monitored (Kralik et al., 2001). 
Cortical sites were selected based on neuroanatomical atlases and 
published microstimulation maps, and localized using stereotaxic 
coordinates, cortical surface features and intraoperative neural map- 
ping. For the squirrel monkey, arrays of 16 electrodes were implanted 
in each of the four areas for a total of 64 electrodes. For the owl 
monkey, arrays of 16 electrodes were implanted in PMd, M1 and S1 
and an array of 32 electrodes was implanted in PPC, for a total of 80 
electrodes. Postoperatively, the monkeys were monitored closely for 2 
weeks and appropriate veterinary care was given to assure safe and 
comfortable recovery. 

The animals’ heads were unrestrained during recording. A Multi- 
channel Acquisition Processor (MAP, Plexon, Inc., Dallas, TX, USA) 
was used to record and discriminate the neuronal signals and to record 
hand position and muscle activity (Nicolelis et al., 1997). A Shape 
Tape™ device (Measurand, Inc., Fredricton, NB, Canada) recorded 
three-dimensional hand position, as described previously (Wessberg 
et al., 2000; Kralik et al., 2001). EMG signals were recorded from the 
skin surface overlying the short head of the biceps, the lateral head of the 
triceps, the lateral head of the deltoid, and from the extensor digitorum 
muscle using pairs of gold disk electrodes (Grass-Telefactor, Astro- 
Medical, Inc., West Warwick, RI, USA). Electrode placement was 
guided by visual inspection and palpation of the muscles. The EMG 
signals were amplified and filtered (SA Instrumentation, Encinitas, CA, 
USA), then passed to the MAP and recorded as analog signals in the 
same data file as the neuronal signals. Each session was videotaped, and 
video records were synchronized with the neural, positional and EMG 
data by a video timer connected to the MAP. Figure 1 depicts EMG and 
hand position data taken from a single trial. Because the movements 
were relatively stereotypical and consisted of a fixed sequence of a 
reaching movement followed by grasping and delivery of the food to the 
mouth, the EMG patterns were replicated (with some variability) from 
one trial to another. Moreover, some muscles were coactivated during 
certain phases of the movement. For example, deltoid was coactivated 
with triceps, with both being active during the early phase of the trials, 
and wrist extensor with biceps, with both active during the later phase. 
Although coactive, these muscles served different functions. 

We could exclude crosstalk as a source of coactivation for several 
reasons. First, coactive muscles clearly differed in the EMG onset and 
fine features of their EMGs, such as EMG burst structure and duration. 
Secondly, spatially nearby muscles (biceps and triceps), which showed 
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Fic. 1. Recorded and predicted EMG activity and hand position. The black 
traces in the upper four panels are single trial surface EMG signals recorded 
from the biceps, triceps, deltoid and extensor digitorum. The black trace in the 
bottom panel shows the position of the monkey’s hand along the front-to-back 
axis (relative to the monkey’s body). EMG activity and hand movement all 
show different time courses of activity. The red traces in all five panels show 
the predictions of these same variables for the trial depicted. Note that in each 
case the model captured the unique time course of the predicted variable. 
Arrows and dotted lines mark the times of behavioral events in the course of the 
trial: Lift, when the hand first lifted from the start position; Tray, when the hand 
reached the piece of food on the tray; Mouth, when the hand reached the 
monkey’s mouth. Biceps, triceps and deltoid depicted are from the same trial, 
while extensor digitorum is from another trial, so the timing of bursts in this 
muscle relative to the other muscles and hand position is only approximate. 


more crosstalk, were not coactivated. Thirdly, crosscorrelation analy- 
sis showed similar levels of zero-lag correlation between coactivated 
pairs as between noncoactivated pairs. To estimate the proportion of 
the signal due to crosstalk, we calculated the square of the maximum 
value of the normalized cross-correlation function (R2,) (Winter, 1991) 
for the coactivated pairs of muscles (triceps and deltoid, biceps and 
extensor digitorum) and for noncoactivated pairs (biceps and deltoid, 
triceps and extensor digitorum). The percentage of the signal due to 
crosstalk was quite low, 0.47 + 0.76% for coactivated pairs and 
0.35 + 0.39% for noncoactivated pairs. The difference between the 
Ry values for the two groups was not significantly significant 
(P = 0.80, paired Student’s ¢-test). 


Analysis of modulations in EMG and neuronal signals 


The continuous records were divided into single trials by examination 
of the videotaped hand movements. Each trial was defined as the time 
period beginning 200 ms prior to the onset of hand movement and 
ending 1000 ms after. This period of time encompassed the entire 
reach, grasp and bringing of the food to the mouth. To analyse changes 
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in EMG activity, an envelope of the EMG signal was calculated by 
conventional means (Soderberg & Kuntson, 2000). EMG activity traces 
were full-wave rectified and smoothed by convolution with a triangular 
kernel of unit area and 75 ms width. The resulting signal was then 
resampled at 100 Hz. The EMG envelope is a conventional estimate of 
the electrical activity in the recorded muscle. Although the EMG 
envelope reflects muscle force, its relation to force is complicated by 
additional factors such as muscle length (Tax et al., 1989; Vander 
Linden et al., 1991; Worrell et al., 2001), velocity (Olney & Winter, 
1985) and muscle contraction history (Poliakov, 1987). 

Peri-event time histograms (PETHs) were constructed relative to the 
EMG onsets of each muscle sampled (see Fig. 2). To determine EMG 
onsets, peaks were first identified in the rectified EMG signals in each 
muscle. EMG onsets were defined as the time, going backward from the 
time of the peak, where activity fell below a threshold. The threshold 
was set by visual inspection of the record for the whole recording 
session and applied universally to all trials. PETHs were composed of 
5-ms bins, which spanned the period from 200 ms before to 200 ms 
after the EMG onset. To assess whether there were significant 
modulations of neuronal firing rate, these PETHs were considered for 
three 150-ms time windows: the 150 ms just prior to EMG onset, the 
150 ms centered at EMG onset and the 150 ms just after onset. We 
assessed the significance of the changes in firing rate during these time 
windows by comparing the firing rates for the bins within each of these 
time windows with the firing rates for the rest of the bins in the PETH 
using the Studentized range criterion (multiple comparisons test). 


Linear regression model for prediction of EMG signals 


A spike density estimate of the neuronal data, initially sampled at 
40 kHz with the times of spike occurrences time-stamped at 1000 Hz, 
was formed by convolution of a train of impulses representing the 
times of spikes with a Gaussian kernel (MacPherson & Aldridge, 
1979), the width of which was inversely proportional to the average 
firing rate of the neuron over the whole session. This was necessary in 
order to generate good estimates of firing rate dynamics for neurons 
whose firing rates could differ by as much as two orders of magnitude. 
The resulting spike density was then resampled at 100 Hz. 

EMG signals were extracted from neuronal activity using a multiple 
linear regression model of the form 


yi(t) = oe Sb, Xing lt — djj,) + et) (1) 


J 


where y,(f) is the activity of muscle i at time ¢, x,(¢ — dj) is the spike 
density of neuron j at time (¢ — dj), dj; is the time lag for neuron-EMG 
pair ij (i.e. the neural activity used for predictions preceded the muscle 
activity being predicted by dj), by is the weight for neuron j and 
muscle i, and ¢é(f) is residual error (Wessberg et al., 2000; Kralik et al., 
2001; Carmena ef al., 2003; Wessberg & Nicolelis, 2004). The 
subscript n enumerates time lags d, between neuron j and muscle i. In 
matrix form, this equation becomes 


y=XBt+s (2) 


where y is a column vector of the activity ofa given muscle from all trials 
in an experimental session, X is a matrix of neural activity from all trials, 
with each column corresponding to a neuron’s spike density at a 
particular lag (i.e. the activity in_X is offset, by a fixed lag time, prior to 
the activity in the corresponding row of y), and f is a column vector of 
the weights. In this representation, each column of X is a predictor 
variable or ‘regressor’. Each regressor contained spike density data 
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sampled at 100 Hz. We considered lags ranging from 10 to 1000 ms, at 
10-ms intervals. Note that regression of eqn (2) captured the relationship 
between neuronal activity and both amplitude and timing of the EMG. 
Timing was captured by the value of the lag parameters (d,)). 

One of the principal pitfalls to be avoided while doing prediction 
using a model with a large number of regressors is overfitting of the 
data (Babyak, 2004). Overfitting occurs when the model begins to fit 
the noise in the data; and it is characterized by high performance of the 
model in fitting the data, but very low performance in predicting a 
novel dataset. We observed overfitting when all the potential 
regressors were included in the model. The mean r of fit (correlation 
coefficient between model output and y for the fitted data) was 0.89, 
whereas the mean r of prediction (correlation coefficient between 
model output and y for data not used in fitting) was 0.49. To combat 
this overfitting, we used a ‘subset selection’ technique, which used a 
reduced set of regressors to maximize prediction performance. We 
used the correlations between the neuronal and EMG signals for each 
neuron—EMG pair at each lag to determine the activity of which 
neurons and which time lags to include in these reduced sets. 
Regressors with high correlation with the EMG provided the best 
signals for prediction, whereas the others could be discarded as noise. 
The reduced set was selected by first ranking the potential regressors 
according to their correlation with the EMG signal. The performance 
of subsets of different size was then evaluated by using n regressors 
with the highest ranks to predict EMG activity. To obtain the r of 
prediction for each set of regressors, the model was first fit using half 
of the trials randomly selected from a given experimental session. This 
model was then used to predict the EMG activity in the other half of 
the trials. The process was repeated using the second set of trials for 
fitting and the first set for predicting. The procedure of randomly 
selecting trials, fitting and predicting was repeated five times, and the 
average r of prediction for that set of regressors was used as the 
measure of prediction quality. After this analysis was performed for all 


Monkey 1 


Biceps Triceps 


the different sized sets of regressors, the set that maximized r of 
prediction was selected as the optimum. 

To quantify prediction performance as a function of neuronal 
sample size, neuron-dropping curves (NDCs; Wessberg et al., 2000; 
Carmena et al., 2003) were constructed. These curves were formed by 
selecting random subsets of n neurons, where n ranged from one to the 
total number of neurons in the dataset, performing subset selection, 
and obtaining the r of prediction for that particular set of n neurons. 
This procedure was repeated 30 times with new sets of n randomly 
selected neurons, and the mean correlation coefficient was entered as 
the appropriate NDC value. 


Results 
Neuronal modulations around EMG onset 


We recorded from multiple neurons simultaneously in M1, PMd, S1 
and PPC of New World monkeys and examined the relationship of the 
neuronal activity in these areas to EMG signals recorded from the 
biceps, triceps, deltoid and extensor digitorum muscles. Two to three 
muscles were recorded per session. In all sessions, both biceps and 
triceps were recorded. On average, 135 neurons per session (147 in 
Monkey 1 and 119 in Monkey 2) were sampled. To assess whether 
modulations in individual neurons were correlated to muscle activity, 
the firing rates of the neurons were examined for changes around the 
time of EMG onset. As seen in Fig. 1, there were prominent bursts of 
EMG activity in all muscles during the course of a single trial. The 
profiles of EMG activity were different from those of the kinematic 
parameters: hand position, velocity and acceleration. For example, 
compare the bottom panel of Fig. 1, which plots hand position along 
the front-to-back axis (relative to the monkey’s body), with the EMG 
activity profiles. In general, in the first part of the trial, as the hand 
moved forward toward the piece of food, the triceps and deltoid were 
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Fic. 2. Examples of neurons from all cortical areas that showed modulation of firing rate relative to onsets of biceps and triceps EMGs. The upper panels depict 
session averages of EMG activity aligned in time relative to the EMG onsets to illustrate that EMG onsets were detected accurately. The lower traces show peri-event 
time histograms of neuronal activity relative to the same EMG onsets for example cells in all cortical areas. Note the variety in the timing and the sign of the firing 


rate modulations. 
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the most active muscles. Their EMG bursts were often accompanied 
by smaller bursts in the biceps and wrist extensor reflecting muscle 
coactivation. Later, as the hand was moving back to the mouth, the 
largest bursts of activity occurred in the biceps and wrist extensor, and 
the triceps and deltoid were less active. The time and amplitude of the 
bursts varied from trial to trial due to differences in target direction and 
speed of reach. 

As an indicator of individual neuronal modulation with the EMG, 
PETHs centered at the onsets of bursts in EMG activity were 
constructed. Figure 2 depicts example PETHs from cells showing 
significant modulations around the onsets of bursts in either biceps or 
triceps in each cortical area of each monkey. Note the heterogeneity of 
the firing rate modulation patterns. Upward and downward modula- 
tions can be seen before, at the time of and after EMG onsets. In our 
neuronal sample, we found no systematic differences between cortical 
areas in the direction or the timing of modulations. Overall, 46% of the 
neurons in Monkey 1 and 54% of the neurons in Monkey 2 were 
found to be modulated around EMG onset in either biceps or triceps, 
the muscles which were recorded in every session. The percentages of 
modulated cells were similar when EMG onsets of different muscles 
were used to center the PETHs. Averaging across the two monkeys 
and the recording sessions, 37, 39, 34 and 33% of cells were 
modulated around EMG onsets in biceps, triceps deltoid and extensor 
digitorum, respectively. Examination of each of the recorded cortical 
areas individually showed that, averaging across monkeys, 41% of 
PMd neurons, 48% of M1 neurons, 47% of S1 neurons and 56% of 
PPC neurons tested showed a significant modulation to EMG onsets. It 
is important to note that this initial analysis did not provide proof that 
neurons were causally or directly related to muscle activity. Rather, it 
estimated the capacity of individual neurons to contribute to EMG 
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predictions for the studied range of movements by virtue of their 
correlations with EMG modulations. The capacity of EMG predictions 
to depict finer variation in the EMG signals was tested by additional 
analyses. 


Prediction of EMG activity 


The predictions obtained using the linear model captured the main 
features of the EMG modulations. Figure 3 illustrates that the profile 
of the EMG modulations for all muscles was predicted for most of the 
trials in an experimental session. The top panels show that the 
predicted muscle activity (red traces) tracked the profile of observed 
activity (blue traces). Not only were the most prominent bursts in 
EMG successfully predicted, but in some cases the model could also 
predict smaller bursts in activity. The bottom panels of Fig. 3 illustrate 
the degree of correlation between actual and predicted signals across 
all experimental sessions for each muscle. The r of prediction values 
averaged across sessions were 0.77 + 0.05 for biceps, 0.71 + 0.08 for 
triceps, 0.74 + 0.01 for deltoid and 0.78 + 0.02 for extensor digitorum 
(mean + 1SD), all of which were highly statistically significant 
(P < 0.001). The extreme values for r of prediction were 0.54 for one 
session in triceps and 0.86 for one session in biceps. For 89% of cases 
r of prediction was above 0.70. Predictions were similar for the two 
monkeys, with average r of predictions for biceps and triceps being, 
respectively, 0.75 and 0.68 for Monkey 1 and 0.80 and 0.73 for 
Monkey 2. 

The prediction performance of the models was highly dependent on 
the number of regressors, as shown in Fig. 4. Regressors were added 
in the order of their strength of correlation with the muscle activity. 
For small numbers of regressors, adding more increased prediction 
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Fic. 3. Comparison of predictions with EMG data. The upper panels show observed and predicted EMG traces in biceps, triceps, deltoid and extensor digitorum for 
all trials in an experimental session. The blue traces are the rectified, smoothed and normalized EMG records, and the red traces are the predictions of the model. 
Note that the model captured the main temporal features of the muscle activity consistently for the entire session. The x-axis is time (in seconds) relative to movement 
onset; the y-axis is in normalized units. The lower panels depict the correlation between observed (x-axis) and predicted EMG amplitude (y-axis) for all experimental 
sessions. The figure depicts a scatter plot of predicted (y-axis) vs. observed (x-axis) amplitude of the rectified, smoothed and normalized EMG trace for all muscles. 
The data shown are from all trials and across all sessions. Best-fit linear regression lines are also plotted. 
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Fic. 4. The effect of subset selection on model prediction performance. As 
regressors are added, r of fits always increases, while r of predictions increase 
at first, then reach a peak and start to decrease, due to overfitting. Using a subset 
selection procedure eliminated this effect of overfitting on prediction perform- 
ance. 


performance, reaching a peak at, on average, about 60 regressors. As 
regressors continued to be added, however, prediction performance 
decreased, due to overfitting. The optimal number of regressors was 
used in the further analyses for each muscle and each experimental 
session. On average, the optimized models used 61 regressors drawn 
from the activity of 44 individual neurons or 33% of the total neuronal 
sample. 


Generalization of EMG predictions 


In addition to predicting the overall profile of EMG activity, our model 
was able to capture the variability in EMG activity from trial to trial. 
The trial-to-trial differences in latency to peak, peak amplitude and the 
overall shape of the EMG envelope could be considerable. The left 
plots in Fig. 5 show the observed EMG envelope (color-coded) for all 
trials in a session, sorted by the onset of the most prominent burst. The 
middle column plots the predicted EMG envelopes. It can be seen that 
the predicted EMG captured the trial-to-trial differences in latency 
because the predicted EMG onsets follow the same course as the 
observed EMG onsets. The third column plots the correlation between 
actual and predicted latency to burst onset for the trials shown. For all 
recorded muscles and in every recording session, this correlation was 
significant (P < 0.001), as was the correlation between observed and 
predicted burst amplitude (P < 0.001). It can also be seen that the 
model captured the differences in timing of EMG activity onset and a 
longer duration of EMG activity in deltoid than in coactivated triceps. 
The model also captured the difference in EMG burst duration 
between extensor digitorum and biceps. 

Along with differences in latency to burst, some trials showed major 
differences in the overall shape of the EMG envelope. Figure 6 
illustrates this kind of difference for a number of trials in a single 
session. The panels show observed (top panel) and predicted (middle 
panel) traces for biceps (orange) and triceps (blue), and a comparison 
of the mean traces (bottom panel). In the left column are the trials that 
showed the most common EMG pattern, having a single, large burst in 
triceps during the reach followed by a burst in biceps when bringing 
the food to the mouth. In the panels of the right column are trials in 
which the EMG traces showed a less typical pattern: a double burst in 
triceps and no burst or a very late burst in the biceps. In these trials the 
monkey took longer to grasp the food. Nonetheless, comparing the 


traces in the bottom panels, it can be seen that the model was able to 
predict the EMG envelope of both biceps and triceps for the trials with 
the double-burst profile. 

Overall, predicted and actual amplitudes of major EMG bursts were 
well correlated for the biceps (r = 0.63), deltoid (r = 0.50) and the 
extensor digitorum (r = 0.60) and marginally correlated for the triceps 
(r = 0.35). All of these correlations were highly significant statisti- 
cally (P < 0.001). In addition, the predicted and actual burst onset 
times in individual trials were also correlated (r = 0.57 for biceps, 
r= 0.43 for triceps, r= 0.62 for deltoid, r = 0.56 for extensor 
digitorum, P < 0.001). 

Another test of generalization of predictions we used was to train 
a model on one set of data and predict another set of data in which 
the movements were different. All of the analysed trials consisted of 
reach and grasp movements, but there were clear differences 
between trials, such as the differences in reach direction and the 
overall pattern of movement (e.g. the single- vs. double-burst pattern 
described above and depicted in Fig. 6). We first sorted the trials by 
direction of reach, predicting trials in which the monkey reached to 
the far left target from models fit with data from reaches to the far 
right target and vice versa. This procedure resulted in a small 
(1.6 + 2.7%) but statistically insignificant decrease in prediction 
performance compared with the case when trials of random 
directions were used to train the model, with prediction of biceps 
falling from r=0.75 to 0.74 (1.1 41.8%) and triceps from 
r = 0.65 to 0.64 (2.1 + 3.5%). 

We also divided trials based on latency to peak and predicted long- 
latency from short-latency trials and vice versa. Again, there was little 
and statistically insignificant difference in prediction performance 
between models with sorted vs. randomly chosen training sets (r of 
prediction decreased 2.4 + 1.4%). The decrease was from r = 0.77 to 
0.75 (2.8 + 1.3%) for biceps and from r = 0.71 to 0.69 (2.1 + 1.6%) 
for triceps. 

Finally, we used the typical and atypical trials depicted in Fig. 6 as 
our two training and testing data sets. For triceps, where in typical 
trials there was a single burst and in unusual trials a double burst, there 
was again a very modest performance decrement from r = 0.77 to 
0.74 (6.4 + 1.3%). For biceps, which had a single burst in the normal 
trials and no burst in the unusual trials, there was a significant drop in 
performance from r = 0.74 to 0.52 (27.4 + 4.4%). This was because 
the variance in biceps activity in the atypical trials was very small, 
hindering the training of a model. 


Contributions to EMG predictions of neurons in different 
cortical areas 


Each of the four cortical areas contributed to EMG prediction to a very 
similar degree. For all muscles, neurons in all cortical areas were 
included in the models with similar frequency. Averaging across the 
monkeys, sessions and muscles, 38% of PMd neurons, 35% of M1 
neurons, 29% of S1 neurons and 30% of PPC neurons were included 
(Fig. 7A). The relative percentages of neurons included in each area 
were also very similar for each muscle in each monkey individually, 
with the exception that for Monkey 2, in all muscles, PPC neurons 
were included less frequently than neurons in other areas (27% vs. an 
average of 42%). This may have been due to the fact that so many 
PPC neurons were recorded (approximately half of the neuronal 
sample in that monkey). 

Even though the neurons from the different cortical areas were 
selected in similar proportions, the neurons from each area may have 
contributed to predicted muscle activity nonuniformly. To test this, we 
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Fic. 5. Prediction of trial-to-trial variability in latency to EMG burst. In the left column, the normalized EMG envelope amplitude is plotted in pseudocolor for all 
trials in a single session. Each muscle had a single prominent EMG burst. The middle column plots the predicted EMG envelope for the corresponding trials. The 
model captures the trial-to-trial variability in the latency to peak of the observed EMG. The right column plots the correlation between latency to peak in the observed 
(x-axis) and predicted EMG (y-axis). For all muscles, this correlation was highly significant (P < 0.01). 


evaluated how strongly the regressors from each cortical area were 
related to muscle activity. Figure 7B shows, by area, the overall 
distributions of the correlation coefficients between muscle and neural 
activity for the regressors selected by the models. None of the 
distributions, which were pooled from biceps and triceps muscles in 
both monkeys and all experimental sessions, was found to be 
significantly different from the others (ANOVA, F = 1.56, P = 0.20). 

Further, the timing of each cortical area’s modulation was analysed 
relative to the EMGs. The distributions of the lags of the EMG relative 
to neurons from each cortical area included in the models are shown in 
Fig. 7C. As in Fig. 7B, the distribution represents models for biceps 
and triceps in both monkeys and all experimental sessions. The 
distributions were similar, and none was significantly different from 
the others (Kruskal-Wallis, y” = 4.09, P = 0.25). In all areas, the 
distributions were strongly skewed towards shorter time lags, with 
34-38% of regressors having lags of 0.1 s or less. Median lag times 
were 0.20 s for PMd, 0.26 s for M1, 0.31 s for S1 and 0.28 s for PPC. 


Prediction quality and neuronal ensemble size 


The prediction performance of different neuronal population sizes 
and different areas was assessed using NDCs (Fig. 7D and E). 
These curves are the average results of 30 regression analyses and 
subset selection procedures (see Methods) using random sets of n 
neurons, for values of n up to the total number of neurons 
recorded. The results are averaged over biceps and triceps for all 
experimental sessions. It is important to note that, unlike the curves 
in Fig. 3 that show performance decreasing as the number of 
regressors is increased, the NDCs in Fig. 7D show that prediction 
performance continued to increase with the total number of neurons 
sampled. This is because the subset selection procedure was 
used for each prediction: with larger samples of neurons, there 
were more regressors for the model to choose from, and 
more information about the predicted variable that could be 
extracted. 
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The top row shows observed EMG traces for biceps (orange) and triceps (blue) together in the same panels. The middle row shows the predicted EMGs using the 
same conventions. The bottom row compares mean observed (dotted lines) and predicted EMG traces. The model captures the double burst pattern in triceps with 
high accuracy, and captures the no burst pattern in biceps reasonably well (e.g. compare orange traces in bottom-left with orange traces in bottom-right panel). For all 
panels, the x-axis is time relative to movement onset and the y-axis is normalized EMG envelope amplitude. 


To compare the prediction performance of the cortical areas 
(Fig. 7E), NDCs were constructed using only cells recorded in a 
given cortical area, with results averaged across both monkeys and all 
experimental sessions. Because of the averaging, the curves extend 
only as far as the least number of neurons recorded in a given area. 
The NDC analysis again showed that neurons in all cortical areas were 
able to predict EMG activity. However, there were small differences 
between the areas. To compare the curves, we calculated the ratio of 
the values of the curve for a particular cortical area to the values of the 
overall NDC, composed of neurons in all areas, across the entire 
length of the curve for the area in question. PMd neurons performed 
13% better than the average of all the neurons together, M1 and S1 
neurons performed 4% better, and PPC neurons performed 10% worse 
than the overall average. These small differences that escaped 
detection by the analysis of Fig. 7B and C could reflect NDCs being 
sensitive to small differences in the percentages of EMG-related 
neurons (Fig. 7A) and population effects such as the correlation of 
activity in different neurons. Thus, when considering all of the 
neurons recorded in each cortical area, we again found that all of the 
areas were able to predict EMG activity, with our sample of PMd 
neurons performing somewhat better than the other areas, and PPC 
neurons performing the least well. 


Discussion 


In this study we found that predictions of EMG activity of arm 
muscles can be obtained from ensemble activity simultaneously 
recorded from M1, PMd, S1 and PPC. The basic features of the EMG 
envelope and its trial-to-trial variability were predicted. Moreover, the 
predictions captured trial-to-trial variations in the amplitude and 
timing of EMG bursts. Using the neuron-dropping procedure we 
demonstrated that prediction performance increased with the size of 
the neuronal population sampled, and continued to rise to our 
maximum population size of about 135 neurons. These results extend 
previous findings that employed neuronal ensemble activity to predict 
hand trajectory in New World monkeys (Wessberg ef a/., 2000) and 
rhesus macaques (Taylor et a/., 2002; Carmena et al., 2003). Carmena 
et al. also used a similar method to predict EMG activity from neural 
activity in multiple cortical areas. In that study, surface EMGs from 
biceps, wrist flexor, wrist extensor and trapezius muscles were 
recorded while rhesus monkeys performed tasks involving moving a 
two-dimensional manipulandum to targets and applying a specified 
amount of force to a grip sensor. Continuously recorded EMG 
envelopes were predicted at levels of performance similar to the 
current study (r = 0.78). Taken together, these results show that this 
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neurons performed least well. 


approach can be used to predict EMG activity from multiple muscles, 
in multiple species, and in multiple behavioral tasks, lending further 
support to the idea that the approach could be used in human 
neuroprosthetic applications. Such applications may require simulta- 
neous prediction of EMGs for a large number of muscles, so an 
important question is how many of the muscles involved in control of 
the arm can be predicted, as well as how many will need to be 
predicted to control a prosthesis. In the current study, activity patterns 
of all muscles recorded were predicted by the algorithm. To explore 
this approach further, larger number of muscles will need to be 
recorded (Park et al., 2000). 

We tested the ability of our prediction algorithm to generalize by 
training our model on one set of data and testing it on another set. 
EMG activity patterns were different in the training and testing 
datasets. The algorithm was able to generalize to different directions of 
reach, to different EMG burst latencies and to nonstandard EMG 
bursting patterns. We noted only one case in which the algorithm did 
not perform well. In this case, there was very little variability in the 
biceps EMG, and the model did not generalize well to the nonstandard 


bursting pattern. This observation suggests that the ability of the 
algorithm to generalize depends on the range of variability in the 
training set. It must be noted, however, that all of the data used to test 
for generalization came from the same task. Therefore, it remains to be 
tested whether this approach can generalize to a wider range of 
movements. Nonetheless, for a neuroprosthetic application, a BMI that 
could generalize well enough to allow patients to feed themselves 
would be of great therapeutic value. 

Subset selection, a novel technique in the BMI literature, allowed 
our model to combat overfitting and significantly improved prediction 
performance, especially in models with a large number of potential 
regressors. In addition, reduction in the number of model parameters 
by subset selection greatly reduced the number of computations 
needed to fit a model, or to compute a predicted EMG value. If 
implemented in an online prediction algorithm, subset selection could 
therefore improve performance in a neuroprosthetic application. 

Looking at the percentage of individual neurons related to EMG, 
their strength of correlation with the EMG, or the percentage and 
quality of neurons included in the models for EMG prediction, we 
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found contributions from the sampled cortical areas of approximately 
equal significance. The NDC analysis did show small (on the order of 
10%) differences between the neuronal samples collected in different 
cortical areas. Our PMd sample performed best, followed by M1 and 
Sl, and then by PPC. Using similar linear regression and NDC 
analysis techniques for data collected in the same species, Wessberg 
et al. (2000) also found that PMd performed better than M1 and PPC 
in predicting continuous hand position of owl monkeys that were 
moving a one-dimensional manipulandum to the left and right. Close 
involvement (either direct or indirect) of PMd in production of 
movements in New World monkeys has been demonstrated in the 
study of Preuss et al. (1996) who showed that forelimb movements 
could be evoked by microstimulation of premotor cortex. By contrast, 
Carmena et al. (2003) found that when rhesus monkeys performed a 
two-dimensional arm movement in which they moved a manipulan- 
dum and then applied a specified amount of grip force to it, M1 
predicted motor parameters better than S1, PMd and PPC. The 
difference between the present results and the results of Carmena et al. 
(2003) may be due to differences between the monkey species. 
Differences in the tasks could be another contributing factor. For 
example, PMd has been shown to be generally less sensitive to limb 
position (Scott et al., 1997) and load (Werner et a/., 1991) than M1, 
which could have affected predictions of motor parameters in the tasks 
using a manipulandum that required precise tracking of hand position. 

The results of this study add to an accumulating amount of evidence 
that encoding of movement parameters is accomplished by distributed 
cortical networks. In a distributed processing scheme, many streams of 
information are being processed in parallel, and an individual neuron 
can simultaneously process more than one type of information 
(Pribram, 1971; Rumelhart et al., 1986; Churchland & Sejnowski, 
1989; Bridgeman, 1992; Chapin & Nicolelis, 1995; Nicolelis, 1998). In 
support of this, Carmena et a/. (2003) reported neurons that simulta- 
neously encoded hand position, velocity and gripping force. In this 
study, cortical neurons were observed that contributed to predictions of 
EMGs in different muscles. This is in agreement with the previous 
studies that showed that cortical neurons which make projections to the 
spinal cord are not purely related to the control of a single muscle (Fetz 
& Cheney, 1980; Shinoda et al., 1981; McKiernan et al., 1998). In the 
current study, we found neurons that carried information relevant to 
EMG prediction in many cortical areas, consistent with the idea of 
distributed processing. We wish to emphasize that by reporting that a 
neuron carried information important for predicting muscle activity we 
do not mean to imply that the neuron only processed information 
related to muscle activity or was directly connected to spinal 
motoneurons. Instead, the presence of EMG-like signals may have 
been only a part of the information processed by the neuron and may 
have occurred by an indirect mechanism. 

There are a number of ways that a neuron which did not carry 
information directly involved in controlling muscles via projections to 
the spinal cord could have provided information to our model. For 
instance, a PMd neuron could carry information about movement 
planning (Weinrich & Wise, 1982), but because that planning 
eventually resulted in movement, and the movement was associated 
with a particular pattern of muscle activation, the planning-related 
activity of that neuron could be transformed by the model into a 
muscle-related signal. Or, an S1 neuron, for example, could receive 
copies of motor commands from M1 (efference copies) in order to 
prepare for the reafferent somatosensory signals resulting from 
movements (Von Holst, 1954; Soso & Fetz, 1980; Chapin & 
Woodward, 1982; Lebedev et al., 1994; Fanselow & Nicolelis, 
1999). This information could also potentially be extracted by the 
model. In addition, M1 neurons correlated with muscle activity could 


be involved in intrinsic processing related to motor computations but 
not themselves participate in sending information to the spinal cord. 
One intriguing possibility is that the distributed processing of motor 
information by neuronal ensembles in the PMd, M1, S1 and PPC may 
be involved in inverse kinematics and inverse dynamics transforma- 
tions. Internal models for movement dynamics appear to be present 
across multiple cortical and subcortical areas (Imamizu et al., 2000; 
Wolpert et al., 1995). Thus, EMG-related neurons may be distributed 
across many areas and in each of them represent certain aspects of 
transformation between kinematic and dynamic variables involved in 
motor control. 

Although the nature of the relationship we have found between the 
neurons from multiple cortical areas and muscle activation may be 
quite complex, this does not diminish the practical significance of 
these results to the development of neuroprosthetics. If a neuron 
contributes to the model’s ability to predict motor behavior or to 
control a neuroprosthetic device, it may not be critical if it is directly 
involved in control of the muscles under normal conditions. In fact, 
once a group of neurons starts to control a mechanical actuator, or 
controls the electrical stimulation of muscles, these neurons in effect 
become the motoneurons for the device or the limb. Previously, we 
suggested that through the operation of a neuroprosthetic device by a 
BMI, that device could actually be incorporated by the neural system 
as if it were a part of the individual’s body (Carmena et al., 2003). 
Likewise, cortical neurons that are made to act as motoneurons could 
adapt through neuronal plasticity to carry out that function. 

Advances have been made in recent years in the development of 
BMIs in which brain activity controls an artificial actuator (Chapin 
et al., 1999; Wessberg et al., 2000; Taylor et al., 2002; Carmena et al., 
2003). For instance, Carmena et al. (2003) have recently shown that a 
robotic device can mimic both reaching and grasping movements of 
primates. The current results suggest that the signals of individual 
muscles that can be extracted from the neural ensemble activity may 
be used to control even more sophisticated and biologically inspired 
robotic devices that can be driven by muscle activation signals. Such a 
system would broaden the range of activities the robotic actuator 
would be able to perform by commanding not only the desired 
position but also the desired stiffness matrix. This is important for 
interacting with objects with different physical characteristics and for 
different purposes. For example, one would want a very different 
stiffness matrix for hitting a baseball with a bat than for painting a 
picture with a brush. In this way, muscle-based prediction could lead 
the way to more versatile neuroprosthetic devices. 
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Rewards are known to influence neural activity associated with both 
motor preparation and execution. This influence can be exerted 
directly upon the primary motor (M1) and somatosensory (S1) cortical 
areas via the projections from reward-sensitive dopaminergic neu- 
rons of the midbrain ventral tegmental areas. However, the neuro- 
physiological manifestation of reward-related signals in M1 and 
S1 are not well understood. Particularly, it is unclear how the neurons 
in these cortical areas multiplex their traditional functions related to 
the control of spatial and temporal characteristics of movements with 
the representation of rewards. To clarify this issue, we trained rhesus 
monkeys to perform a center-out task in which arm movement 
direction, reward timing, and magnitude were manipulated in- 
dependently. Activity of several hundred cortical neurons was 
simultaneously recorded using chronically implanted microelectrode 
arrays. Many neurons (9-27%) in both M1 and S1 exhibited activity 
related to reward anticipation. Additionally, neurons in these areas 
responded to a mismatch between the reward amount given to the 
monkeys and the amount they expected: A lower-than-expected re- 
ward caused a transient increase in firing rate in 60-80% of the total 
neuronal sample, whereas a larger-than-expected reward resulted in 
a decreased firing rate in 20-35% of the neurons. Moreover, re- 
sponses of M1 and S1 neurons to reward omission depended on 
the direction of movements that led to those rewards. These obser- 
vations suggest that sensorimotor cortical neurons corepresent 
rewards and movement-related activity, presumably to enable 
reward-based learning. 


reward | motor cortex | multichannel recording | primate | prediction error 


eward-based learning is a fundamental mechanism that al- 

lows living organisms to survive in an ever-changing envi- 
ronment. This type of learning facilitates behaviors and skills that 
maximize reward (1). At the level of motor cortex, synaptic modi- 
fications and formation of new synaptic connections that accompany 
learning have been reported (2). The reinforcement component of 
these plastic mechanisms is partly mediated by dopamine (DA), a 
key neurotransmitter associated with reward (3-5). DA inputs to 
cortical areas, including primary motor (M1) and somatosensory 
(S1) areas, arise from the midbrain (6-8). Recent studies demon- 
strating the presence of a higher concentration of DA receptors in 
the forelimb over the hind-limb representation in rodent motor 
cortex (9), and impairment of motor skill acquisition resulting from 
blocking these DA receptors in the forelimb representation (10), 
point to the role of DA receptors in learning skilled forelimb 
movements. In agreement with this line of reasoning, animals with a 
rich repertoire of skilled movements, such as monkeys and humans, 
show a disproportionate increase in the density of DA receptors in 
the cortical motor areas: M1, premotor, and supplementary motor 
cortex (6, 11, 12). 

Although evidence points to the involvement of M1 and 
S1 circuitry in reward-related learning and plasticity (1, 13, 14), it 
is still not well understood how M1 and S1 neurons process 
both reward and motor information. A longstanding tradition in 
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neurophysiological investigations of M1 and S1 was to attribute 
purely motor and sensory functions to these areas. Accordingly, the 
majority of previous studies have focused on the relationship of 
M1 and S1 spiking activity with such characteristics as movement 
onset (15), movement direction (16, 17), limb kinematics (18, 19), 
muscle force (20-22), temporal properties of movements (23, 24), 
and progression of motor skill learning (25). Fewer studies have 
examined nontraditional representations in M1 and S1, such as 
spiking activity associated with rewards (26-31). Representation of 
reward in M1 was provoked by several studies that, in addition to 
motor-related neural activity, examined neuronal discharges in 
these areas that represented motivation, reward anticipation, and 
reward uncertainty (30-33). Although these studies have confirmed 
that M1 neurons multiplex representation of reward with the 
representation of motor parameters, a number of issues need 
further investigation. Particularly, it is unclear whether reward- 
related neuronal modulations in M1 (31, 33) represent rewards 
per se, their anticipation, and/or the mismatch between the actual 
and expected rewards. Furthermore, the relationship of these 
reward-associated signals with the processing of motor information 
is poorly understood. Understanding of these interactions will help 
elucidate mechanisms underlying reward-based learning (34). 

To this end, we have developed an experimental paradigm in 
which reward and characteristics of movements were manipu- 
lated independently: (i) Movements and reward were separated 
by two different delays, (ii) multiple movement directions were 
examined for their impact on reward, and (iii) unexpected var- 
iability in reward amounts were introduced to produce their 
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mismatch to the animal’s expectation of reward. Neuronal en- 
semble activity was recorded chronically in rhesus monkeys using 
multielectrode arrays implanted in both M1 and S1. These ex- 
periments revealed a widespread representation of reward antici- 
pation in M1 and S1 neuronal populations. Additionally, many 
M1 and S1 neurons responded to unexpected changes in reward 
magnitude [i.e., represented reward prediction error (RPE)]. 
Moreover, neuronal responses to RPEs were directionally tuned; 
they varied according to the direction of the movement planned by 
the animal. Taken together these results indicate that predictive 
encoding of reward is robust in both M1 and S1 and that reward- 
related signals interact with neuronal directional tuning in senso- 
rimotor cortex. As such, reward-related neuronal modulations in 
the monkey primary sensorimotor cortices could contribute to 
motor learning (35). 


Results 


Our central goal was to test whether M1 and S1, at both the single- 
cell and the ensemble level, can mediate reward-based learning. 
More specifically, we sought to measure cortical neuronal activity 
that encoded reward anticipation, unexpected changes in reward 
presentation, and modulation of reward-associated signals by arm 
movements. For this purpose, we used a task in which animals 
moved a realistic image of a monkey arm (avatar arm) on a 
computer screen using a hand-held joystick. In any given trial, the 
avatar hand had to be moved and placed over a circular, colored 
target and held over it for 3 s. A passive observation version of the 
task (Fig. 1C) was also implemented. In this version, a computer 
controlled the movement of the virtual arm while the monkey, 
whose limbs were restrained from moving, was rewarded for 
simply observing the task displayed on the computer screen. The 
color of the target informed the monkeys about the time when a 
juice reward would be delivered. 

In early-reward trials, cued by a blue target, reward was de- 
livered 1 s after target acquisition, whereas in late-reward trials, 
cued by a red target, reward was delivered after 2 s (Fig. 1C). In 
the joystick control task, after receiving the juice reward (0.66 mL, 
500-ms pulse) monkeys continued to maintain the avatar arm at 
the target location for the remaining hold period. Holding still at 
the target location was encouraged by small juice rewards (50 ms, 
0.03 mL) that were delivered periodically (every 500 ms) during 
the entire 3-s hold period. Changes in position during the hold 
time precluded the remaining rewards. All of the monkeys learned 
the requirement to hold the virtual arm still. In a majority of the 
trials (>90% in monkeys M and R and 80-85% in monkey C) the 
monkeys did not move their hands. The SD of joystick position 
during the hold time was 0.014 cm in monkey M, 0.06 cm in 
monkey C, and 0.025 in monkey R. 

Because the total hold time period (3 s) was the same irrespective 
of the early or late reward delivery, the reward rate was maintained 
irrespective of trial sequence. We expected that this homogeneity of 
the reward rate would result in similar movement patterns for both 
trial types even though the rewards were given at different times 
after the movement was completed. Indeed, the reaction times (RT; 
Fig. SLC) and movement times (MT; Fig. S1B), which are proxies 
for motivation levels, were similar for both trial types. The early- 
(blue trace in cumulative histograms) and late- (red trace) reward 
trials were not significantly different [P > 0.05, Kolmogorov-Smirmnov 
(KS) test, n = 3 monkeys]. The same held true in a majority of 
behavioral sessions (24/25 sessions in monkey M, 12/13 in monkey 
C, and 7/7 in monkey R). It was found that monkeys only weakly 
favored early reward over the late reward as inferred by a small 
difference between the numbers of attempted early-reward trials 
and late-reward trials (Fig. SLA, monkey M: 0.6%, monkey C: 2.4%, 
and monkey R: 1%). In summary, although monkeys had a slight 
tendency to initiate early-reward trials more often, there were no 
significant differences in the movement parameters. 
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Fig. 1. Experimental setup and schematic of the paradigm. (A) Rhesus mon- 
keys were trained to perform a center-out task using a custom-made hand-held 
joystick. Moving the joystick displaced an avatar monkey arm on the computer 
monitor. (B) Neural data were obtained from M1 neurons in three monkeys 
(monkeys M, C, and R) and S1 neurons in one monkey (monkey M) using 
chronically implanted arrays (black and brown rectangles). (C) Experiment 
schematic: Trial began when monkeys positioned the arm within the center 
target. Following fixation (FT), a circular target (T), red or blue, appeared at a 
distance of 9 cm from the center in one of eight target locations. Monkeys were 
trained to react quickly (RT) and acquire the target by moving the avatar arm 
(MF, movement forward). After reaching they held the arm at the target lo- 
cation for 3 s. A juice reward (R) was delivered either after 1 s (early reward) or 
after 2 s (late reward) into the hold time. Target color (blue vs. red) cued the 
monkey regarding reward time (early, cyan vs. late, magenta). Small juice re- 
wards were given every 500 ms during the 3-s hold period (HT, hold time). At 
the end of the hold period, monkeys returned the arm to the center (MB, 
movement back). The next trial began after a brief intertrial interval (ITI). (Inset) 
The avatar monkey arm relative to a target-size disk. Note the cursor (0.5-cm 
yellow circle) has to be completely within the target to count as a successful 
“reach.” In Exp. 2, in 10% of the trials out of the total a surprise reward (SR) was 
given to the monkey before the target appeared. 


Neuronal Modulations During Different Task Epochs. A total of 
485 neurons were recorded in the three monkeys (M1, monkey M: 
n = 125, monkey C: n = 176, monkey R: n = 184; S1 from monkey 
M: n = 96). Several representative neurons with different kinds of 
neuronal responses elicited during the task are illustrated in Fig. 24. 
The data are separated into joystick control sessions (Fig. 24, Left) 
and passive observation (Fig. 24, Right), and early- (cyan) and late- 
(magenta) reward conditions. The associated arm position (joystick 
control session), the virtual arm position (passive observation ses- 
sion), and the electromyographies (EMGs) from biceps and triceps 
(passive observation session) are displayed above the perievent time 
histograms (PETHs). Some neurons were modulated by the posi- 
tion of the arm, that is, neuronal firing increased (e.g., RHA1_005a) 
or decreased (e.g., RHA3_008a) when the arm moved away from 
the center. Some neurons were modulated by the arm movement 
velocity: The neuronal firing increased as a function of the tan- 
gential velocity (e.g., RHA3_012b). Some neurons were modulated 
by arm position and reward, meaning that neuronal firing increased 
as the arm moved away from the center and decreased till the end 
of reward delivery epoch and increased right after, and, finally, 
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Fig. 2. Neuronal modulation during task epochs. (A) Example neurons from a joystick control session (Left) and passive observation session (Right) from Exp. 
1. (Left) The changes in the mean (z-scored) arm position (at the top), followed by binned (50 ms), z-scored firing rate (PETH) of example neurons, all aligned 
to onset of hold time. (Right) A passive observation session; the EMG activations are displayed at the top. EMGs were measured from left (IT) and right triceps 
(rT), and left (IB) and right biceps (rB). Then, the virtual arm position is plotted. Neurons tuned to position, velocity, reward, or a combination of those is 
displayed next. Bands around the mean represent +1 SE. Data are color-coded based on the time of reward: early (blue) and late (red). Time of reward is 
indicated in the upper panels by color-coded downward-pointing arrows. Note that the arm is stationary during the hold period. Weak muscle activations 
were seen around the time of movements in the passive observation session. (B) Example neurons plotted in the same format as in A; however, the y axis 
depicts the absolute firing rate modulation. The lower panels show spike raster plots. (C) PETH of the example neurons shown in B aligned to the time of 


reward onset. y axis depicts the normalized firing rate modulation. 


spiking activity decreased again when the arm moved back to the 
center (e.g., RHA2_030b). Some arm velocity-related neurons also 
maintained spiking activity at a level higher than baseline till the 
end of the reward delivery epoch (e.g., RHA1_023b; see also 
RHAL 022c in Fig. 2B). Some neurons did not respond to position 
or velocity but increased their spiking activity till the end of the 
reward delivery epoch (e.g., RHA3_009b). Note that the neurons 
in a column, with different response properties, were all recorded 
simultaneously during the same session. 

Fig. 2 B and C show two example M1 and S1 neurons whose 
spiking activity distinguished early from late reward. PETHs shown 
in Fig. 2B are aligned on hold onset and those in Fig. 2C are aligned 
on reward onset. The data are separated into joystick control ses- 
sions (Left) and passive observation (Right) and early- (cyan) and 
late- (magenta) reward conditions. The associated arm/hand posi- 
tion and licking frequency (joystick control session) and the virtual 
arm position (passive observation session) are displayed above the 
PETHs as in Fig. 2A. Spike rasters are also included in Fig. 2B. 
Although the arm position was stable (RHA1_023b in Fig. 24 and 
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RHAL 022c in Fig. 2B), and the EMGs (seen in Fig. 2A) did not 
show any significant muscle contractions around the time of reward, 
the M1 and S1 neurons exhibited differences in firing rate before 
reward onset (P < 0.05, two-sided f test). 


Reward-Related Modulation in the M1/S1 Neural Population. The 
population average PETH for monkeys M, C, and R during 
joystick control are displayed in Fig. 34 and during passive ob- 
servation conditions for monkeys M and C in Fig. S24. Pop- 
ulation PETHs, centered on hold onset, for the early- (cyan) and 
late-reward trials (magenta) show that the neuronal ensemble 
average firing rate is very high for movement (t < 0), except in 
monkey M for passive observation. The average spiking activity 
decays during the hold time up to the time of reward. Early-trial 
neural activity can be distinguished from late-trial activity, es- 
pecially in monkey C (both joystick control and passive obser- 
vation) and monkey M (joystick control). In all of the monkeys 
the average PETHs were not very different for the early and late 
trials before the onset of the hold period. 
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Simply averaging the spiking activity across the neural population 
may conceal some differences. Thus, to examine whether the neu- 
ronal ensemble can distinguish early- from late-reward trials, before 
the early reward (0-1 s of the hold period) we used a machine 
learning algorithm-the support vector machine (SVM). For that, we 
used a 500-ms-wide moving window along the trial time axis, while 
separate SVM classifiers were trained for each window position to 
decode the trial into early vs. late rewards. Different window du- 
rations, step sizes, and algorithms were also tested (Materials and 
Methods). To compare the representation by different cortical areas 
of the early- vs. late-reward conditions, the classifier was also trained 
on different pools of neurons (ipsilateral and contralateral M1, S1, 
etc.). Irrespective of whether the neurons were selected from the 
ipsilateral or the contralateral M1, the classifier could clearly dis- 
sociate early- from late-reward trials in the initial 1-s hold period 
preceding any of the rewards [shaded window in Fig. 3B, monkey C: 
contralateral M1, receiver operating characteristic (ROC): 0.77 + 
0.11; ipsilateral M1, ROC: 0.72 + 0.09; P < 0.05, ROC post-SVM]. 
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Fig. 3. Representation of reward by the neuronal population. (A) Population 
firing rate: The firing rate of each neuron was computed, z-scored, and av- 
eraged to plot the population PETH for each monkey for a joystick control 
session. The associated confidence bands are also shown (+1 SE). (B and C) 
Neuronal ensemble-based classifier analysis. Proportion of correct classification 
of trials, denoted by decoding accuracy, according to reward time (1 or 2 s) is 
shown on the y axis. The time aligned to hold onset is shown along the x axis. 
The classifier was trained using all of the contralateral M1 neurons (purple) or 
the ipsilateral M1 neurons (green). Note that the classification was significantly 
better (P < 0.05, bootstrapped ROC) than chance even before reward was 
delivered (at 0-1 s of hold period, shaded gray box) when using ipsilateral or 
contralateral M1 neurons (B) or $1 neurons (C). 
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S1 neurons carried much less information to separate early- from 
late-reward trials because decoder performance was lower than that 
obtained with ensembles of M1 neurons (Fig. 3C, monkey M: M1, 
ROC: 0.6 + 0.08; S1, ROC: 0.54 + 0.04; ROC post-SVM). 

As an additional test, we examined passive observation sessions 
where the animals did not produce any movements of their arms. 
Even in the passive observation task the SVM classifier could 
distinguish between early- and later-reward trials, based on the 
neuronal population activity recorded during the initial hold time 
period (0-1 s; Fig. S2B, monkey C: ROC: 0.58 + 0.06; monkey M: 
0.62 + 0.07). These classification results suggest that reward time 
classification was not due to differences in arm movement patterns 
between the early- and late-reward conditions. 

The association between the target color and neural representa- 
tion of the expected time of reward (early vs. late) was reinforced 
with practice. If the difference in M1/S1 neural activity between 
early- and late-reward trials reflected this association, then predic- 
tions of reward timing based on neural ensemble activity should 
improve with practice. Consistent with this hypothesis, during the 
first session we observed that M1 neural ensemble activity, produced 
during the initial hold time period (0-1 s), did not clearly distinguish 
between early- vs. late-reward trials (Fig. S2C, orange trace; monkey 
C: ROC: 0.51 + 0.1 and monkey M: 0.54 + 0.07). However, after a 
few sessions of training (joystick control sessions: monkey C, nine 
sessions and monkey M, eight sessions), M1 neural ensemble activity 
could distinguish early- from late-reward trials (Fig. S2C, blue trace; 
monkey C: ROC: 0.77 + 0.11 and monkey M: 0.62 + 0.08). As a 
control, the target color information was withheld in the last session, 
by changing it to a neutral color (white). Immediately, the classifier 
performance dropped to chance levels during the initial hold time 
(0-1 s) (Fig. S2C, black trace for monkey C; ROC: 0.5 + 0.1). 

In summary, the neuronal population-based classifier analysis 
could distinguish the early- from late-reward trials before early 
reward. The improvement in classifier output with training sug- 
gests a learned association between the target color and the time 
of reward, which may be strengthened over time. 


Modulation in Neuronal Activity Due to a Mismatch from Expected 
Reward Magnitude. We next asked whether M1/S1 neurons re- 
sponded to unexpected changes in reward magnitude. To test this, 
in a small fraction (20%) of randomly interspersed trials reward was 
withheld (NR, no-reward trials). In these trials, the animals received 
a lower-than-expected amount of juice, referred to as a negative 
RPE (34, 36). In another subset of trials (20%), the animals re- 
ceived more than expected juice (0.9 mL, 750-ms pulse; DR, 
double-reward trials), referred to as a positive prediction error. In 
the remaining 60% of the trials the expected amount of reward was 
delivered (0.46 mL, 350-ms pulse; SR, single-reward trials). 

When reward was withheld, negative RPE, a brief period of 
increased neuronal firing activity was observed in the PETHs of 
some neurons (RPE in Fig.44, Right), whereas others, recorded 
during the same session, did not show this increase in spiking 
activity (Fig. 44, Left). Fig. 4B shows the PETH of another ex- 
ample neuron (RHA_012a), along with the associated raster plot 
and licking frequency (see SJ Materials and Methods for more in- 
formation on licking frequency estimation). This M1 neuron dis- 
played a significant increase in spiking activity when reward was 
omitted (cyan and magenta RPEs). More example neurons from 
monkeys M and R are displayed in Fig. 4C, Top and Fig. S3. These 
neurons exhibited a significant increase in spiking activity in re- 
sponse to reward omission (two-sided f test: P < 0.05). Such an 
increase in neuronal activity was recorded in many M1 and 
S1 neurons (monkey M, 113/221 neurons and monkey R, 38/ 
189 neurons; two-sided ¢ test: P < 0.05), shortly after the potential 
reward time [monkey M: 143 + 5.6 ms (SD), m = 113 cells and 
monkey R: 165 + 5.2 ms (SD), m = 38 cells]. As shown in Fig. 4C, 
Bottom (magenta trace), the average activity of the M1/S1 neuronal 
population also captured this transient neuronal excitation. 
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When the reward magnitude was doubled-positive RPE a sig- 
nificant reduction in spiking activity was observed. Neurons in Fig. 
4C, Top and Fig. S3 display this reduction in spiking activity (green 
trace; two-sided ¢ test: P < 0.05) when the juice delivery epoch 
extended beyond the regular period of 350 ms. This effect, 
which was significant in many neurons (monkey M, 55/221 cells 
and monkey R, 30/189 cells), could also be captured by the 
average population activity of M1/S1 neuronal ensembles (Fig. 
4C, Bottom, green trace). 

In summary, a majority of M1 and S1 neurons responded to 
both positive and negative RPE by producing opposing changes 
in firing rate modulation: an increase to signal a reward with- 
drawal and a reduction for a reward increase. 


ol) iii’ 


M1 and S1 Neurons Multiplex Target Direction, Arm Velocity, and Reward 
Information. In a center-out arm movement task, M1/ S1 neurons 
are tuned to changes in the location of the target, direction of 
movement, and arm velocity, among other parameters. To assess 
the relative strength of the reward-related modulation in the same 
M1/S1 neuronal spiking activity, a multiple linear regression analysis 


A Hand velocity Hand velocity 


was carried out by including reward magnitude, along with cursor 
velocity and target location, as regressors. This analysis was carried 
out independently in seven different, 400-ms-wide trial epochs [Fig. 
5A: baseline (B), fixation (F), target (T), movement forward (MF), 
hold (H), reward (R), and movement back (MB)]. Single-reward 
and no-reward trials were included in one model; double- and 
single-reward trials were part of another model. 

As expected, variability in firing activity in many M1/S1 neurons 
could be accounted for (P < 0.05) by changes in cursor velocity 
(monkey M: M1, 88%; B = 0.18 + 0.006. S1: 84%; B = 0.17 + 0.008. 
Monkey R: M1, 59%; B = 0.08 + 0.0053) and target location (Fig. 
5A; monkey M: M1, 73%; B = 0.067 + 0.016. S1:68%; B = 0.06 + 
0.019. Monkey R: M1, 71%; B = —0.15 + 0.008), especially during 
the movement phases (MF and MB in “cursor velocity” and “tar- 
get” panels in Fig. 5A). However, reward magnitude significantly 
affected M1 neural activity in the reward epoch (R in “reward” 
panel of Fig. 5A): A sizeable population of M1 and S1 neurons 
robustly responded to a negative RPE (Fig. 54; monkey M: M1: 
82.6%; B = —0.15 + 0.0069. S1:61%; B = —0.13 + 0.01. Monkey R: 
M1, 63.4%. B = —0.08 + 0.0058). 
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Fig. 4. Neuronal modulation due to mismatch in the expected reward magnitude. (A) Plots showing hand movement trace and the neural activity of ex- 
ample neurons from the same session when a reward is omitted—a no-reward trial. The plot format is the same as that of Fig. 2A. Note the transient increase 
in spiking activity, indicative of RPE, after the potential reward onset time for the neuron in the right but not in the left panel. (B) Another example neuron, 


along with the absolute firing rate, licking frequency, and spike raster during a 


no-reward trial. The plot format is adapted from Fig. 2B. (C) (Top) Example 


neuron from monkey M (Right) and monkey R (Left). The normalized neural activity is plotted with respect to time of reward for single-reward trials (black), 
double-reward trials (green), and no-reward trials (magenta). (Bottom) The average population neural activity. Bands around the mean represent +1 SE. The 
horizontal lines below the abscissa span the duration of the single reward (black, 350 ms) and the double reward (green, 700 ms). Note that when reward is 
withheld a transient increase in neural activity is observed, whereas when more than the expected reward is delivered a reduction in activity is witnessed. The 
arrows (downward-pointing, red) indicate when the neural activity was significantly different from the regular reward condition. 
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During reward delivery, when the flow of juice began to diminish, 
monkeys licked at the juice spout. A potential concern was whether 
the licking movements were represented in the neuronal modula- 
tions observed during the reward epoch. Although neural record- 
ings were performed in the forearm areas of M1 (monkeys M and 
R) and S1 (monkey M), licking movements could be potentially 
represented by the M1/S1 neurons. In fact, a small percentage of 
neurons (<5%) showed spiking activity that corresponded to licking 
frequency. Licking frequency was determined using a custom 
methodology and added as a regressor in subsequent analysis. We 
observed that some neurons (“licking frequency” panel in Fig. 5A; 
monkey M: M1 = 13%, S1 = 7.5%; monkey R: 17%) were weakly 
tuned to changes in the frequency of licking (M1: 6 = 0.0032 + 
0.0032; S1: B = 0.01 + 0.002, monkey R: 0.05 + 0.007). More im- 
portantly, changes in reward magnitude continued to account for 
most of the variability in M1/S1 neuronal spiking activity, as they did 
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before introducing licking frequency as an additional regressor 
(monkey M: M1, 82.6%; B = —0.15 + 0.0069. $1:61%; B = —0.13 + 
0.01. Monkey R: M1, 63.4%; B = —0.08 + 0.0058). 

To isolate effects related to reward anticipation, the multiple 
regression analysis was performed by including reward antici- 
pation as an additional regressor, along with other confounding 
variables, such as cursor velocity, target location, licking fre- 
quency, reward magnitude, and so on. Because reward antici- 
patory activity is expected to increase as the time of reward 
approaches (e.g., RHA3_009b in Fig. 2 A and B), the “time 
elapsed after target appearance” was included as the regressor 
that represents anticipation. By running this modified regression 
model we obtained the fraction of neurons significantly modu- 
lating (P < 0.05) their neural activity in accordance with the time 
elapsed after target onset. The fraction was represented as a func- 
tion of time. We observed that, as the time of reward approached, a 
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Reward-related neural activity is multiplexed along with other task relevant parameters. (A) Epoch-wise multiple regression analysis. Each panel shows fraction 


of neurons whose variability in spiking activity could be explained by changes in cursor velocity (panel 1), target position (panel 2), reward (panel 3, single reward vs. no 
reward), reward anticipation (panel 4), surprise reward (panel 5), and licking frequency (panel 6), using a multiple regression model (P < 0.05), in different 400-ms-wide 
epochs during a trial: B, before fixation; F, fixation; T,target onset; MF, moving (forward) to target; H, hold period; R, reward delivery; and MB, moving back to center. 
(B) Spatial tuning of reward-related neural activity. Normalized firing rate of example cells is plotted for eight different arm movement directions. Average + SEM 
(smaller than the markers) of the neural activity is shown for different epochs: during movement (300-700 ms after target onset, black) and after the expected time of 
reward (200-600 ms, green) for the no-reward trials. (C) The directional tuning of all of the neurons for the movement and reward epochs (in no-reward trials) was 
assessed using multiple regression. The directional tuning of all of the neurons is captured by the scatter plot. The model assessed neuronal firing rate with respect to 
different target positions along the x axis and y axis. By plotting the respective (x and y) coefficients (Coeff.x and Coeff.y, respectively) in 2D space, each neuron’s 
preferred direction (angle subtended by the neuron with the abscissa) and tuning depth (radial distance of the neuron from the center) were determined. Tuning during 
the movement (black dots) and after the expected time of reward (200-600 ms, green) for the no-reward trials is shown. (D) The histograms show the number of 
neurons that underwent a change in spatial tuning from the preferred direction. Directional tuning of many neurons changed by 180 + 45°. 
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greater fraction of cells reflected anticipation. For example, in 
monkey M the number of M1 anticipation-related neurons in- 
creased as the time of reward approached (Fig. 5A, “reward an- 
ticipation”): 0% around target appearance, 4% during movement, 
25% during the hold time, and 27% around the time of reward. 
S1 neurons had a similar trend with the exception of a noticeable 
dip around the time of reward: 0% around target appearance, 1% 
during movement, 14% during the hold time, 9% around the time 
of reward, and 25% toward the end of hold time. In monkey R the 
effect of reward anticipation was weaker but nevertheless a signif- 
icant population of M1 neurons showed the same trend: 0% around 
target appearance, 3% during movement, 4% during the hold time, 
and 9% around the time of reward. These results confirmed that 
M1/S1 spiking activity reflected the time elapsed from target onset. 
Notably, we did not see any significant changes in the representa- 
tion of the other regressors (velocity, target location, etc.), which 
suggests that the time-dependent changes are specific to reward 
anticipation and not related to any movements. 

To aid better visualization of the increase in reward anticipatory 
activity, trials were separated into early- and late-reward trials, and 
the regression analysis was performed in every time bin leading up 
to the reward. More specifically, we performed an independent 
regression on every 100-ms time bin, using the average of 100 ms 
of data. In this analysis we also included all of the previously used 
regressors: instantaneous velocity, target position, reward out- 
come, time after target appeared, surprise reward, and licking 
frequency. Trials for this analysis were aligned to target onset, 
instead of to each trial-relevant epoch. Because the reward time 
was staggered, +2 SD of the distribution was determined for both 
early- and late-reward trials and indicated in the plots (black 
vertical lines, Fig. S44). By running this continuous regression 
model, in monkey M, as expected from the previous analysis, 
gradually more and more M1/S1 neurons significantly modulated 
their firing rate as the time of reward approached (Fig. S44). 
Importantly, the time course of this increase in the number of M1/ 
S1 cells tuned to reward plateaued around the time reward was 
delivered. As a result, the time course was faster for the early- 
reward condition compared with the late-reward condition. A 
similar trend, albeit weaker, can be seen for monkey R also. Be- 
cause monkey C did not participate in Exp. 2, and because licking 
movements were not monitored for monkey C, the regression 
model for this monkey only included target position, velocity, and 
reward anticipation. However, a similar trend could also be seen 
for reward anticipatory activity in monkey C despite the utilization 
of a reduced model. Interestingly, in monkey C some neurons 
were observed to be modulated right after target appearance, 
which may be indicative of the prospective reward. In all, these 
results reinforce the notion that M1/S1 neurons also modulate 
their firing rate as a function of reward anticipation. 

Having examined the impact of reward omission (negative 
RPE) on spiking activity in M1 and S1, a similar multiple linear 
regression analysis was used to study the influence of an un- 
expected increase in reward (positive RPE). Several neurons 
modulated their firing rate significantly (P < 0.05) by positive 
RPE, although the effects were weaker than for negative RPE 
(monkey M: M1, 45.5%; B = —0.05 + 0.0037. $1:22%; B = —0.05 + 
0.005. Monkey R: M1, 35.5%; B = —0.037 + 0.0043). We antici- 
pated that a more salient positive RPE would be generated if the 
monkeys were rewarded when they least expected it. To do so, ina 
few (10%) randomly chosen trials animals received juice (surprise 
reward in Fig. 1C) while they waited for the trial to begin. Al- 
though some neurons (~10%) in monkey M modulated their fir- 
ing rate to the unexpected reward, a much larger fraction of 
neurons responded in monkey R (“surprise reward” panel in Fig. 
5A; M1, 35%; B: —0.04 + 0.003). To assess the strength of the 
neural activity elicited by surprise reward we examined the re- 
gression coefficients generated by the regression model (continu- 
ous regression was performed on every 100 ms, as mentioned 
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previously). By doing so, we observed that in monkey M, despite 
the fact that only 10% of the neurons significantly responded to 
the surprise reward, these cells showed a clear increase in firing 
modulation strength (Fig. S4B). Both M1 (6 = 0.023 + 0.002) and 
S1 (6 = 0.016 + 0.003) neurons showed an increase in activity 
around the time of the surprise reward. In monkey R, a similar 
trend was observed, with the coefficient of strength increasing to 
reflect surprise reward (B = 0.025 + 0.003). 

In summary, our results suggest that, in addition to target lo- 
cation and cursor movement velocity, RPE, reward anticipation, 
and, to a small extent, licking movements contributed to the 
spiking activity of the M1 and S1 neurons. M1/S1 neurons also 
modulated their firing rate as a result of both negative and positive 
RPEs. The effects of negative RPEs on M1/S1 neurons seemed to 
be stronger and more widespread than those of positive RPEs. 


Reward-Related Modulations in the Spatial Tuning of Sensorimotor 
Neurons. It is well known that the firing rate of M1/S1 neurons var- 
ies in a consistent way according to arm movement direction (black 
trace in Fig. 5B), a property known as neuronal directional tuning. 
Thus, as expected, the normalized firing rate of the neuron 6J4_012b 
from monkey R shows clear directional tuning (Fig. 5B, Left, black 
trace, P < 0.05, bootstrap): a higher firing rate for the target located 
at 90°, therefore referred to as the “preferred” direction, and a lower 
firing rate for the target at 270°, the “nonpreferred” direction. An 
example neuron from monkey M (Fig. 5B, Right, RHA1_012b) is 
also directionally tuned (P < 0.05, bootstrap). To examine di- 
rectional tuning in the neuronal population, a multiple regression 
model (Materials and Methods) was fitted to every neuron’s spiking 
activity when the arm was moving to the target (MF, movement 
forward epoch in Fig. 5A). The model assessed neuronal firing rate 
with respect to different target positions along the x axis and y axis. 
By plotting the respective (x and y) coefficients in 2D space, each 
neuron’s preferred direction (angle subtended by the neuron with 
the abscissa) and tuning depth (radial distance of the neuron from 
the center) were determined. The black dots in Fig. 5C and Fig. S5 
represent the directional tuning of individual neurons. These graphs 
indicate that many neurons were directionally tuned (monkey M: 
206/220; monkey R: 167/198 neurons P < 0.05, bootstrap; monkey 
M: quadrant 3; monkey R: quadrant 2). 

We then asked whether the same neurons exhibited directional 
tuning around the time of reward. For that, a multiple regression 
model was fitted to every neuron’s average spiking activity in the 
last 400 ms, before reward delivery began. The directional tuning of 
every neuron was plotted in Fig. S5A (blue dots). This analysis 
revealed that the directional tuning, calculated at the time of re- 
ward onset, was correlated with the tuning during the movement 
epoch in monkey M (R = 0.7; P < 0.001; n = 220). The tuning was 
weaker in monkey C (R = 0.27; P < 0.001; n = 176) and monkey R 
(R = 0.18; P = 0.002; n = 184). These results suggest that M1/ 
Slneurons were weakly tuned to the target location at the time of 
reward onset. This result is consistent with the observations from 
Fig. 5A (target panel) that show neural activity modulated by target 
location when the target was visible (till the end of hold epoch). 

We then measured directional tuning when the reward was 
omitted. As shown in Fig. 5B (green trace), both neurons illus- 
trated showed strong directional tuning (P < 0.05, bootstrap). 
However, in both cases these neurons’ spiking activity was sup- 
pressed in the preferred direction (90° in 6J4_012b and 180° in 
RHA1_012b). To further examine directional tuning in the same 
M1/S1 neuronal population, a multiple regression model was 
fitted to every average neuronal spiking activity during the first 
400 ms of the potential reward epoch in these no-reward trials. 
Fig. 5C shows the preferred direction and tuning depth for each 
M1 and S1 neuron (green dots and squares) during this epoch. 
These graphs indicate that many neurons underwent a significant 
change in their directional tuning, from the movement to the 
reward epochs (monkey M: quadrant 3-1; monkey R: quadrant 
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2-4). To assess whether these changes in M1 neuronal preferred 
direction were significant, a nested regression model was 
designed and tested (Materials and Methods). This analysis 
revealed that many neurons in monkey R (M1: n = 154/198, P < 
0.05, bootstrap nested regression) and in monkey M (M1: n = 68/ 
126; S1: n = 33/94, P < 0.05, bootstrap nested regression) un- 
derwent a significant change in preferred direction during this 
transition, from movement to reward epochs. Indeed, a fraction 
of these neurons (monkey R, 48%; monkey M: M1, 46%; S1: 
30%) exhibited a change in directional tuning on the order of 
180 + 45° (Fig. 5D). These results indicated that, when reward 
was omitted, a large fraction of M1/S1 neurons exhibited sup- 
pression in firing activity in their preferred direction. 

Next, we tested whether these changes in directional tuning could 
be influenced by the return arm movement produced by the ani- 
mals. The return arm movement was offset by at least 1 s from the 
beginning of the reward delivery epoch in late-reward conditions 
and by 2 s in early-reward conditions. Further, during the postre- 
ward hold time, small rewards kept the monkeys motivated to hold 
the hand at the target. Moreover, the animals delayed the return 
movement (300-500 ms) following reward omission. The task de- 
sign and the behavior would therefore minimize the influence of 
return arm movement on the neural activity at the time of reward 
delivery. However, we also ran a multiple regression analysis to 
compare neuronal directional tuning around the time of return 
movement with that around the time of the potential reward, in a 
no-reward trial (Fig. SSB). The directional tuning during the return 
movement (cyan dots in Fig. S5B) was not correlated with the 
tuning observed during the no-reward epoch (green dots; replicated 
from Fig. 5B) in monkey M (R = —0.05; P = 0.28; n = 220) but 
negatively correlated in monkey R (R = —0.35; P < 0.001; n = 184). 
The return movement had no influence on the neural tuning at the 
time of reward for one monkey, whereas for the second monkey a 
negative relationship between the neuronal activity during these two 
epochs was found, which by itself may not account for the firing rate 
modulations around the time of reward. 


Discussion 


In the present study we manipulated reward expectation using 
explicit instruction of reward timing by color cues. This experi- 
mental setting allowed us to study the representation of temporal 
intervals related to rewards by M1 and S1 neurons, as well as the 
representation of the reward, and the integration of this in- 
formation with the traditional M1 and S1 neuronal firing modu- 
lations, such as encoding of movement direction and velocity. We 
observed that (i) more than 9-27% of M1/S1 neurons exhibited 
anticipatory neuronal activity related to reward timing, which was 
distinct enough to differentiate reward times that were 1 s apart, 
(ii) and modulated their neural activity as a function of errors in 
reward expectation, and (iii) M1/S1 neurons corepresented reward 
and motor parameters. 


Reward-Related Activity in Sensorimotor Neurons. Modulation in 
neural activity was observed in many M1/S1 neurons around the 
time of reward, which we have interpreted as representing both 
expectation of reward and mismatch in reward expectation. These 
findings are consistent with reward anticipatory modulations pre- 
viously seen in premotor cortical neurons (30, 32) and in motor 
cortical neurons (33) (see SJ Discussion for more information). 
However, given that many of these M1/S1 neurons are involved in 
planning of arm movements, a potential concern is that the mon- 
keys could be moving their arms, loosening their grip on the joy- 
stick, or performing other unmonitored hand movements, which 
could be misinterpreted as reward-related modulation. However, in 
our task design any arm movement produced during the hold time 
would preclude the animal from receiving the intended reward. To 
comply with this constraint, all three monkeys learned to hold 
their arms steady. This was confirmed by inspection of the joystick 
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position, during the 3-s hold time, which revealed that the arms of 
the monkeys remained stable on most trials during the hold time. 
Despite a steady position, one could still attribute the observed 
M1/S1 neural firing modulation to a change in arm or body pos- 
ture, which led to no change in the arm end point. However, 
because the monkeys were operating a spring-loaded joystick, 
adjusting their arm/body posture while keeping the joystick sta- 
tionary would be extremely demanding and logistically quite an 
unfeasible task for the animals. Notwithstanding these mitigating 
factors, we have conducted passive observation sessions, where 
monkeys did not operate the joystick at all but simply watched the 
task being performed on the screen. In these sessions, the mon- 
key’s hands were well-restrained to the sides of the chair, while 
EMGs were recorded to control for any inadvertent muscle con- 
tractions. A slight increase in EMG was recorded from the (long 
head of) triceps and the biceps as the virtual arm moved; however, 
no EMG modulations were observed around the time of reward. 
Under these conditions of complete absence of overt arm move- 
ments, clear reward-related M1/S1 neuronal firing modulations 
occurred in several neurons (e.g., RHA2_030b and RHA3_009b 
in Fig. 2). Furthermore, examination of neuronal firing rate 
modulations demonstrated that the spiking activity of some neu- 
rons represented arm position and velocity, but not reward-related 
activity, whereas other simultaneously recorded neurons displayed 
reward-related activity. If the reward-related activity was influ- 
enced by arm movements, this finding would be unlikely. Fur- 
thermore, a few M1/S1 neurons, recorded simultaneously during 
the same session, displayed only reward-related activity. 

Finally, we identified licking-related neurons and the firing 
rate modulations in these neurons followed the changes in lick- 
ing frequency. To determine the changes in neural activity that 
were correlated with licking movements, we have used a gener- 
alized linear model analysis (37-39). From this analysis we 
learned that spiking activity of only a small percentage of neu- 
rons was related to licking frequency (<15%), whereas a much 
higher number of our recorded M1/S1 neurons represented arm 
movements (>80%) and changes in reward (>50%). These 
findings are consistent with the fact that our multielectrode ar- 
rays were originally implanted in the so-called forearm repre- 
sentation of M1/S1. Based on all these controls, we propose that 
our results are totally consistent with the interpretation that the 
observed M1/S1 neuronal firing rate modulations described here 
were related to reward expectation, rather than emerging as the 
result of some unmonitored animal movement. 


RPE Signals in M1/S1. Following “catch” trials when a reward was 
withheld, a transient increase in neural activity was observed in 
many M1 and S1 neurons. This observation is consistent with a 
recent study (31) that reported an increase in firing in M1 and 
dorsal premotor (pMd) neurons when reward was not delivered 
in one monkey. We have replicated those findings in our study 
and further extended them by demonstrating that M1 and 
S1 neurons reduced their firing rate for a higher-than-expected 
reward. Altogether, these results support the hypothesis that 
M1 and S1 encode bidirectional RPEs. Our findings are also 
consistent with those obtained in an imaging study (40) and a 
recent electrocorticography study (41) in which RPE-related 
activity was detected in sensorimotor brain regions. 

Neuronal response to both positive and negative RPEs, in- 
dicated by distinct bidirectional changes in firing rates, was initially 
reported for midbrain DA neurons (34, 36, 42). Subsequently, bi- 
directional RPE signals have been observed in many other brain 
areas (43-47). The widespread nature of RPE signals may suggest a 
common general mechanism involved in learning from outcomes. 
Our results suggest that this mechanism may also exist in the 
monkey primary motor and sensory cortical areas as well. 

Although the role of DA in reward processing is rather com- 
plex and still a subject of debate (48, 49), one line of research has 
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implicated the phasic DA response, typically seen in midbrain 
DA neurons, as the mediator of RPEs (36, 50-52). Typically, 
reinforcement-based learning models have used prediction er- 
rors to account for behavioral plasticity (53-55). Recent studies 
that looked at plasticity in the motor system, particularly re- 
garding the role of reward in motor adaptation (13, 56) and fa- 
cilitation of movement sequences (57), have suggested that 
M1 neurons may play a crucial role in this process. The fact that 
we observed here that M1 and S1 neurons exhibit bidirectional 
prediction error signals reinforces this hypothesis significantly. 
Interestingly, a significant fraction of M1/S1 neurons, at the time 
of reward omission, exhibited directional tuning that was distinct 
from the one displayed by them during the arm movement epoch. 
Changes to the M1 neuron directional tuning, and hence, to the 
cell’s preferred direction, has been observed previously, as a 
function of task context (58-60) or even during the time duration 
of a trial (61, 62). During a typical trial, no tuning could be de- 
tected before the target appeared; strong neuronal tuning emerged 
only after the target appeared and remained throughout the period 
the target was acquired, until the end of the hold period. However, 
when reward was withheld, many M1/S1 neurons changed their 
preferred directions by 180 + 45°. We speculate that these changes 
are related to reward omission and may serve to facilitate reward- 
based learning. However, it still remains to be tested whether these 
reward-related modulations affect future M1/S1 neural activity. 


Distributed Encoding in M1/S1 Neurons. Based on studies carried 
out in the last couple of decades, M1 neuronal activity has been 
correlated with several kinetic (20-22, 63, 64) and kinematic 
parameters (16, 17, 19). Even though many of these movement 
parameters are correlated with one another, in general one line 
of thinking in the field purports that cortical motor neurons are 
capable of multiplexing information related to different param- 
eters (65-68), including reward (65, 69, 70). Simultaneous re- 
cording from several M1/S1 neurons in our study has shown that 
many of these neurons are associated with some aspect of 
movement—position, velocity, and so on—whereas a subset of 
the neurons also seem to be receiving reward-related signals. 
This latter group of M1/S1 neurons are perhaps the ones that 
displayed both movement and reward-related activity. These 
findings are consistent with those from recent studies that have 
examined neurons in M1 and pMd responding to arm position, 
acceleration, and reward anticipation (31). 

In conclusion, we observed that M1/S1 cortical neurons si- 
multaneously represent sensorimotor and reward-related mod- 
ulations in spiking activity. According to this view, M1 and 
S1 ensembles can be considered as part of a distributed network 
that underlies reward-based learning. 
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Materials and Methods 


All studies were conducted with approved protocols from the Duke University 
Institutional Animal Care and Use Committee and were in accordance with 
NIH guidelines for the use of laboratory animals. Three adult rhesus macaque 
monkeys (Macaca mulatta) participated in this study. More information re- 
garding animals and the implants are available in S/ Materials and Methods. 


Behavioral Task. The task was a modified version of the center-out task. A 
circular ring appeared in the center of the screen. When monkeys held the 
avatar arm within the center target, a colored (red/blue) circular target ring 
appeared in the periphery, in one of eight evenly spaced peripheral locations. 
Monkeys had to then move the arm to place it on the target and hold it there 
for 3 s. A juice reward was delivered after 1 s (early reward) or after 2s (late 
reward). The target color cued the time of reward (blue, early and red, late). 
In Exp. 2, in 20% of trials reward was omitted (no reward). In another 20% the 
reward was doubled (double reward). Further, in a small subset of randomly 
chosen trials (10%) a surprise reward (gated by a 350-ms pulse) was delivered 
during the fixation period. More information on the task design can be found 
in S! Materials and Methods. 

Neural activity was recorded using a 128-channel multichannel acquisition 
processor (Plexon Inc.) (71). All of the analyses were performed in MATLAB 
(The MathWorks, Inc.). 


PETH of Neuronal Activity. The recorded action potential events were counted 
in bins of 10 ms, smoothed using a moving Gaussian kernel (width = 50 ms) 
for illustration purposes. Details regarding normalization procedures can be 
found in S/ Materials and Methods. 


Classifying Trials Based on Early vs. Late Reward. To investigate the strength 
of information about the time of reward provided by a single neuron, or 
provided by the entire population, a decoding analysis was performed based 
on a linear discriminant analysis algorithm or SVM algorithm, using the 
MATLAB statistics toolbox (see S/ Materials and Methods for details). 


Multiple Linear Regression. To model every neuron’s changes in firing rate as 
a function of task contingencies (Fig. 5A) a multiple regression analysis was 
carried out with the following general linear models (GLM; see S/ Materials 
and Methods for details). 


Spatial Tuning. The normalized mean firing rate was determined during the 
movement epoch, the hold time epoch, reward epoch, and return movement 
epoch. The z-scored firing rate in each epoch was fit for each neuron with a 
GLM (see S/ Materials and Methods for details.) 
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Abstract 


Principal components analysis (PCA) was used to define the linearly dependent factors underlying sensory information 
processing in the vibrissal sensory area of the ventral posterior medial (VPM) thalamus in eight awake rats. Ensembles of up to 
23 single neurons were simultaneously recorded in this area, either during long periods of spontaneous behavior (including 
exploratory whisking) or controlled deflection of single whiskers. PCA rotated the matrices of correlation between these n neurons 
into a series of n uncorrelated principal components (PCs), each successive PC oriented to explain a maximum of the remaining 
variance. The fact that this transformation is mathematically equivalent to the general Hebb algorithm in linear neural networks 
provided a major rationale for performing it here on data from real neuronal ensembles. Typically, most information correlated 
across neurons in the ensemble was concentrated within the first 3-8 PCs. Each of these was found to encode distinct, and highly 
significant informational factors. These factor encodings were assessed in two ways, each making use of fact that each PC 
consisted of a matrix of weightings, one for each neuron. First, the neurons were rank ordered according to the locations of the 
central whiskers in their receptive fields, allowing their weightings within different PCs to be viewed as a function of their position 
within the whisker representation in the VPM. Each PC was found to define a distinctly different topographic mapping of the 
cutaneous surface. Next, the PCs were used to weight-sum the neurons’ simultaneous activities to create population vectors (PVs). 
Each PV consisted of a single continuous time series which represented the expression of each PC’s ‘magnitude’ in response to 
stimulation of different whiskers, or during behavioral events such as active tactile whisking. These showed that each PC 
functioned as a feature detector capable of selectively predicting significant sensory or behavioral events with far greater statistical 
reliability than could any single neuron. The encoding characteristics of the first few PCs were remarkably consistent across all 
animals and experimental conditions, including both spontaneous exploration and direct sensory stimulation: PC1 positively 
weighted all neurons, mainly according to their covariance. Thus it encoded global magnitude of ensemble activity, caused either 
by combined sensory inputs or intrinsic network activity, such as spontaneous oscillations. PC2 encoded spatial position contrast, 
generally in the rostrocaudal dimension, across the whole cutaneous surface represented by the ensemble. PC3 more selectively 
encoded contrast in an orthogonal (usually dorsoventral) dimension. A variable number of higher numbered PCs encoded local 
position contrast within one or more smaller regions of the cutaneous surface. The remaining PCs typically explained residual 
‘noise’, i.e. the uncorrelated variance that constituted a major part of each neuron’s activity. Differences in behavioral or sensory 
experience produced relatively little in the PC weighting patterns but often changed the variance they explained (eigenvalues) 
enough to alter their ordering. These results argue that PCA provides a powerful set of tools for selectively measuring neural 
ensemble activity within multiple functionally significant ‘dimensions’ of information processing. As such, it redefines the ‘neuron’ 
as an entity which contributes portions of its variance to processing not one, but several tasks. © 1999 Elsevier Science B.V. All 
rights reserved. 
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1. Introduction 


One of the central concepts of contemporary neuro- 
* Corresponding author. science is that major brain functions are executed 
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through the joint actions of ensembles of neurons (Erick- 
son, 1968; Churchland, 1989). This mode of computation 
may have similarities to parallel distributed processing 
(PDP; Rumelhart et al., 1986a), in which information is 
not localized within individual neurons, but instead is 
distributed across neuronal populations. Thus, no single 
neuron is necessary for any one computation, yet each 
neuron can participate in a large number of computa- 
tions. Computations involve resolving information em- 
bedded in patterns of correlated activity among large 
populations of afferent inputs. 

Assuming that such distributed processing occurs in 
real neuronal populations, how might it be studied? 
According to theory, information which is distributed 
across a neuronal ensemble should be manifested as 
patterns of correlated activity among the neurons. Unfor- 
tunately, it is very difficult to resolve the underlying 
structure of information contained within large numbers 
of pairwise correlations between simultaneously recorded 
neurons. As an alternative, this paper examines the use 
principal components analysis (PCA) for analysis of real 
distributed neuronal networks. PCA is a classical tech- 
nique for obtaining an optimal overall mapping of 
linearly dependent patterns of correlation between vari- 
ables (e.g. neurons). PCA provides, in the mean-squared 
error sense, an optimal linear mapping of the ‘signals’ 
which are spread across a group of variables. These 
signals are concentrated into the first few components, 
while the ‘noise’, i.e. variance which is uncorrelated across 
variables, is sequestered in the remaining components. 
PCA has been used extensively to resolve temporal 
patterns in neurophysiological recordings (McClurkin et 
al., 1991; Kjaer et al., 1994). Here, PCA was used to map 
the neuronal ensemble information in up to 23 simulta- 
neously recorded neurons in the vibrissa-sensory region 
of the ventral posterior medial (VPM) thalamus of awake 
rats, during spontaneous behaviors (including ex- 
ploratory whisking) and controlled deflection of single 
whiskers. 

A further rationale for using PCA to map patterns of 
neuronal correlation is that it is known to be mathemat- 
ically equivalent to the general Hebb algorithm for 
learning in linear neural networks (Oja, 1982, 1989, 1992; 
Bourlard and Kamp, 1988; Baldi and Hornik, 1989; 
Sanger, 1989; Hertz et al., 1991; Hrycej, 1992). Most self 
organizing artificial neural network models utilize some 
variant of Hebbian learning, in which synaptic strengths 
are modified according to the temporal correlation of the 
synaptic inputs with post-synaptic neuronal activation. 
Since post-synaptic activation is normally caused by 
increased synaptic input, the Hebb rule increases synaptic 
strengths according to their temporal correlation with 
other inputs. 

The ultimate aim of this study was to record activity 
of neuronal ensembles at the thalamic level of the 
somatosensory system, and then to use PCA to evaluate 


the mapping of sensory information, providing clues as 
to how thalamic signals are transformed in their transmis- 
sion through the cortical circuitry. For this purpose it was 
necessary to obtain recordings of neuronal activity during 
long periods of spontaneous behavior, hopefully to 
obtain a representative sample of the normal repertoire 
of information handled by these neurons. While several 
investigators have demonstrated that experience depen- 
dent factors can influence patterns of sensory mapping 
in the neocortex, no study has yet employed actual 
simultaneous recordings to evaluate how the sensory 
information might be transformed by Hebb-like learning. 
For this purpose, PCA is advantageous in that it provides 
a mathematically well understood technique for mapping 
the informational factors comprising linearly dependent 
patterns of covariance in a neuronal ensemble. On the 
other hand, independent components analysis (ICA) is 
more appropriate for identification of nonlinear and/or 
independent patterns of covariance (Laubach et al., 
1999). 


2. Materials and methods 
2.1. Recording procedures 


Complete methods for the simultaneous many-neuron 
recording techniques are discussed in detail elsewhere 
(Shin and Chapin, 1990; Nicolelis et al., 1993, 1995; 
Nicolelis and Chapin, 1994). Briefly, a multi-channel unit 
recording/discrimination system (Plexon Inc, Dallas, TX) 
single neuron action potentials were recorded through 
arrays of 16 microwire electrodes (25 or 50 um diameter, 
Teflon insulated stainless steel; NB Labs, Dennison, TX) 
implanted across the whisker representation in the VPM 
in Long-Evans (Hooded) rats. Recording experiments 
commenced approximately 1 week following surgery. 
Once in the recording chamber, a wiring harness was 
plugged onto the previously implanted headstage. The 
multi-neuron recording system allowed simultaneous 
amplification, bandpass filtering, window discrimination 
and computer storage of spike-times from large numbers 
of single neurons. Approximately 90% of the microwire 
electrodes yielded at least one discriminable waveform 
(signal/noise ratio at least 5). Online single unit discrim- 
ination used digital signal processors (DSPs) incorporat- 
ing both voltage—time window and principal component 
clustering waveform discrimination algorithms. The sam- 
ple experimental data sets utilized here included up to 23 
neurons recorded simultaneously in the VPM thalamus. 

Receptive fields (RFs) were quantitatively character- 
ized by mechanically displacing single whiskers using a 
computer controlled vibromechanical actuator. Eight to 
20 different facial whiskers were stimulated 300-600 
times apiece (3° deflections, 0.1 s step pulses, delivered 
at 1 Hz), yielding a highly quantitative data base for 
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evaluation of these neurons’ sensory properties. Such 
data obtained from the same neurons under awake 
and/or anesthetized conditions yielded a center RF for 
each neuron. After the awake animals were highly 
habituated to the whisker stimulation procedure they 
rested quietly without moving. 

Recordings were also obtained during 20- to 60-min 
periods of spontaneous behavior, which included 
episodes of rest, exploratory whisking, grooming and 
locomotion. During recordings in awake, freely moving 
animals, the rat’s behavior was continuously monitored 
and stored on video tape. A Lafayette Super-VHS 
video analysis system was used for field-frame analysis 
of these video tapes (1/60th s resolution). Synchroniza- 
tion of the videotape with experimental recording data 
was achieved by using a 100 Hz output from the data 
acquisition computer (Motorola VME delta system) to 
update a Thalner video counter-timer (Ann Arbor, 
MI), which produced a time-stamp with 10 ms resolu- 
tion, displayed on each fieldframe. 

Analysis of recorded data, including the PCA al- 
gorithm described below, was carried out using soft- 
ware developed by JKC to run on a Motorola VME 
delta system. Further statistical and graphical analyses 
were carried out using the CSS-Statistica package 
(Tulsa, OK) on principal components (PCs). 


2.2. Statistical techniques 


2.2.1. Covariance, cross-correlation and 
cross-correlograms 

Single neuron discharge was quantized by integrating 
over time bins ranging from 5 to 500 ms (with 10 ms as 
a Standard). For statistical analysis, each neuron consti- 
tuted a ‘variable’, and its spike count for each bin 
(generally over the whole experiment) was a ‘sample’. 
First, correlation or covariance matrices were calcu- 
lated. Statistical significance of these correlation coeffi- 
cients was calculated by the formula, 


r=tS, 
where ¢ is Student’s f-test, and 
s. 1—r? 

(n— 2) 


where n is the number of data samples. 


2.2.2. Principal components analysis (PCA) 

The theory of principal components states that every 
symmetrical covariance or correlation matrix relating p 
random variables X1, X2,...,Xn can be transformed 
into particular linear combinations by rotating the ma- 
trix into a new coordinate system. This rotation is 
produced by multiplying each of the original data vari- 
ables by their appropriate weighting coefficients. For 
each component, these weights comprise a vector called 


an eigenvector, and the variance ‘explained’ by its 
eigenvalue. The original matrix is rotated such that the 
axis defined by the first principal component (PC1) is 
aligned in the direction of greatest variance, hence 
maximizing the eigenvalue. To obtain the second com- 
ponent (PC2) the matrix is rotated around the PC1 axis 
to obtain a second eigenvector which again contains the 
greatest possible amount of remaining variance. This 
procedure is repeated until a set of N orthogonal (un- 
correlated) components is obtained, arranged in de- 
scending order of variance. In this transformation, 
none of the information contained within the original 
variables is lost, and the derived components can be 
statistically manipulated in the same way as the original 
variables. Moreover, the transformation is useful be- 
cause most of the significant total variance (i.e. corre- 
lated neuronal information) is concentrated within the 
first few uncorrelated PCs, while the remaining PCs 
mainly contain ‘noise’ (i.e. uncorrelated neuronal infor- 
mation). The first few PCs not only provide a simpler 
and more parsimonious description of the covariance 
structure, they also concentrate the information which 
is normally spread across multiple variables (neurons) 
into a single, more statistically useful ‘factor’. 

The squares of these weighting coefficients represent 
the correlations of each variable (neuron) with the PC. 
Since each component is apportioned some fraction of 
the total variance of the neuron, the sum of squared 
coefficients across each row must equal 1.0. A further 
constraint is that the sums of squared coefficients for 
each component must also equal 1.0. Thus the total 
variance contained in the ensemble can be represented 
either in terms of the combination of neurons (i.e. 
across rows in the eigenvector matrix), or the combina- 
tion of principal components (i.e. across columns). 


2.2.3. Characterization of statistical error 

Although PCA is a distribution free method it is 
sometimes useful to assess the statistical significance of 
the eigenvalues and principal component coefficients 
obtained. The following formulas were used here for 
calculation of S.E. of eigenvalues, 


s(A) = 4), x ja (4) 
n—1 


and principal component coefficients: 


5(byj) = | jan 7 Dal S Ala urea) (5) 


where 7 is the number of samples, 4, is the eigenvalue 
of component h, b,,; is the jth coefficient of the Ath 
component, and p is the total number of components. 

These formulae apply to principal components analy- 
ses carried out on covariance matrices. However, since 
correlation matrices are equivalent to covariance ma- 
trices calculated from standardized data, the S.E. can 
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be considered appropriate when standardized data are 
used to construct a covariance matrix. These formulas 
are technically correct except when used for data with 
small numbers of samples and for severely non-Gaus- 
sian distributed data. The typical situation for the 
simultaneous neural recordings here is to obtain a very 
large (10° to 2 x 10°) number of samples, with near- 
Poisson distributed data. As shown in Fig. 1, these data 
converge on normality with increasing sample integra- 
tion times. 


2.2.4. Interpretation of PCA results 

Since PCA involves successive rotations of a covari- 
ance matrix, the configuration of each derived PC is a 
function of the previous (lower numbered) PCs. Thus, 


the PCs may not each represent a single factor underly- 
ing the population covariance, but instead may provide 
rules for separating between such factors. PCA is not, 
therefore, the method of choice for separating between 
completely independent factors, but is an excellent tech- 
nique for defining a multidimensional mapping scheme 
for linear representation of a set of interdependent 
factors. As such, PCA should be ideal for analyzing 
processing of sensory information which normally in- 
volves interdependent factors, such as movements of 
stimuli over continuous receptor surfaces. 


2.2.5. Using PCA to construct eigenfunctions 
After their derivation here, the PCs were then used to 
create eigenfunctions by using the PC weights to 
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Fig. 1. Eigenvalue rotation of a two-neuron cluster plot. (A) Scatterplot showing the distribution of joint activity states in an ensemble of two 
neurons (1 and 7) recorded simultaneously in the PrV nucleus. (Both of these were among 47 recorded across multiple levels of the somatosensory 
system.) The position of each point depicts the firing rates of these two neurons over a single 500-ms time interval in a continuous experiment 
lasting 1800 s. The line through this point cluster shows their linear regression. Their correlation coefficient is 0.61 (P.€0.01). Since the coordinates 
of these points are integers, many overlap. (B) Scatterplot showing the same data after standardization by subtracting the mean and then dividing 
by the S.D. This removes excessive biasing of neurons with higher firing rates. (C) Half-normal probability plot showing that integrated spiking 
data converges on a normal distribution. The solid line shows the expected distribution of points in a perfectly normal (Gaussian) distribution. 
Points show the standardized scores from neuron 7 which were used to construct Fig. 3B. (D) Scatterplot shows the same data from neurons | 
and 7 after axis rotation using PCA. The best linear fit through the original points shown by the regression line in Fig. 1B is now oriented perfectly 
along (i.e. is ‘explained’ by) the first principal component (PC1), with the second principal component explaining the ‘noise’ in the perpendicular 
axis. The coordinates of the points in this new space defined by components | and 2 were calculated as weighted sums (i.e. the dot product) of 
the original standardized scores (from Fig. 1B). See text for principal component weights used to achieve this rotation. 
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weight-sum the time integrated data stream from each 
of the original neurons. Each eigenfunction is therefore 
a continuous time series containing an analog value for 
each time bin within the experiment. Typically, these 
time bins are chosen to be the same as the time 
integrals used to calculate the original correlation ma- 
trix used for the PCA. As shown in the equation below, 
each point Y,, in an eigenfunction is defined as the 
linear combination of the standardized, time integrated 
spike counts recorded during that time bin from the 
original neurons, X,, multiplied by the appropriate 
eigenvector weights Bp. 


Y, = eX = eX) + 65%. + ...€,1X, 
Y, = e| x= CX + €y9X> + 1 €yrX, 


Dp 
Y= 1X = €1,X1 + Ca, X7 + «Cp Xp 


where ei are eigenvectors, and e, are eigenvector 
weightings. 

In this case X; is the spike count of neuron i within a 
particular time bin. Though Y, is commonly weighted 
by the eigenvalue 4, this is unnecessary for our applica- 
tions. Also, when correlation matrices are used as input 
for the PCA it is necessary to first standardize X, by 
subtracting the mean and dividing by the S.D. The 
eigenfunctions constructed using this technique consti- 
tute composite variables which, in statistical terms, can 
be treated much like the original variables (neurons). 
Since they are constructed from summations of 
weighted and standardized data, the eigenfunction mag- 
nitudes can be thought of in terms of ‘population’ firing 
rates, with units in Hz, but unrelated to actual firing 
rates of single neurons. In the figures below, eigenfunc- 
tions are used like traditional neurophysiological data, 
in stripcharts and peri-event averages. 


3. Results 
3.1. Data sets 


This investigation utilized data obtained from a series 
of experiments in which ensembles of single neurons 
were simultaneously recorded through microwire elec- 
trode arrays implanted at different levels of the trigem- 
inal somatosensory system in awake Long-Evans 
(Hooded) rats. A total of 707 neurons were recorded at 
various levels of the trigeminal somatosensory system in 
20 rats during 497 different recording experiments rang- 
ing from 5 to 60 min in duration. Of these, 481 neurons 
were recorded in the ventral posteromedial (VPM) tha- 
lamic nucleus, specifically in the subregion which repre- 
sents the mystacial whiskers. Of these, 127 VPM 
thalamic neurons in eight animals (ranging from eight 
to 24 neurons/animal) were recorded under experimen- 
tal conditions meeting acceptable criteria for inclusion 


in this study. These conditions required that at least 
eight well discriminated single neurons be simulta- 
neously recorded in the VPM thalamus of an awake rat 
over at least one 20- to 40-min period of spontaneous 
exploratory whisking behavior, and also over a number 
of discrete whisker stimulation experiments sufficient to 
quantitatively define the RFs of each neuron. 

These animals were typically first recorded during a 
long period of spontaneous exploratory behavior in 
which the animal used its whiskers to explore objects in 
the experimental chamber. Post-hoc analyses of syn- 
chronized videotape records of this behavior typically 
show that rats spend about 50% of their time engaging 
in spontaneous whisking movements, especially includ- 
ing active sweeping of the mystacial whiskers tactile 
objects. In subsequent experiments under awake and 
anesthetized conditions the RF properties of the same 
neurons were quantitatively measured using a com- 
puter-controlled vibromechanical actuator to repeti- 
tively displace single whiskers or multi-whisker bundles. 
Thus, these experiments measured the responses of 
trigeminal somatosensory system neuronal ensembles to 
sensory stimulation either delivered passively to single 
whiskers or obtained actively through exploratory 
movement of the same whiskers over objects. The data 
sets were thus ideal for defining neuronal population 
information (using eigenfunctions) within somatosen- 
sory neuronal ensembles in the context of active tactile 
exploration. 


3.2. Example of analysis 


In a typical experimental protocol, spiking data from 
23 well discriminated VPM neurons were simulta- 
neously recorded over a single experimental day. First, 
the RFs of each neuron (i.e. their principal whiskers) 
were defined by aural monitoring of unit activity. Next, 
the 23 neurons were recorded during a 1821.49 s period 
of spontaneous behavior that included several long 
episodes of exploratory whisking. Subsequently, nine 5- 
to 15-min experiments were conducted to measure the 
same neurons’ responses to controlled vibromechanical 
stimulation of each of nine single whiskers. 

In post-hoc analysis, a correlation matrix between 
these 23 neurons was first calculated from the data 
recorded during the spontaneous exploratory period. 
This matrix revealed positive correlation coefficients (r) 
ranging from 0.0054 to 0.39. Despite the relatively low 
r values, they are highly statistically significant (S.E.M. 
0.00354) because of the large number of samples 
(72 859, using 25-ms time integrals). Thus r > 0.011 was 
the confidence limit for a correlation significance at the 
P <0.001 confidence level. In this matrix of 242 ((n— 
1)2;) correlation pairs, all but one r exceeded zero by at 
least this confidence limit. Moreover, the average r in 
this matrix was 0.137, ic. 12.45 times the confidence 
limit. 
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3.3. Using PCA to map covariance structure 


The ‘eigenvalue decomposition’ utilized by PCA in- 
volves rotating the covariance matrix to define a new 
set of orthogonal axes oriented in directions of greatest 
covariance. As shown in Fig. 1, this can be visualized as 
a rotation of a scatterplot in N-dimensional space in 
which each point defines the state of an N-neuron 
ensemble within a particular time interval. To illustrate, 
Fig. 1A shows a 2D scatterplot which depicts the 
correlation between the time integrated activities of two 
simultaneously recorded neurons in the VPM (unit | vs 
unit 7). Fig. 1A was constructed by quantizing the 1821 
s experiment into 3642 bins (using 500-ms time integrals 
for illustration purposes). The spike counts of the two 
neurons within each bin are plotted as a point (vector) 
in an X—Y space, such that the point’s position repre- 
sents the ‘functional state’ of the two-neuron ensemble. 
The r-value of these neuron’s correlation was 0.61, 
which is highly significant (P<0.001). This high positive 
correlation can be visualized by the preponderance of 
points lying along the 45° line in Fig. 1A. 

Though PCA can utilize either covariance or correla- 
tion matrices, use of the former was discontinued be- 
cause the PCA results tended to be dominated by the 
variance of a few rapidly firing neurons. To more 
accurately measure the co-activity of different neurons, 
this study utilized correlations, which are equivalent to 
covariances between standardized variables. Fig. 1B 
shows the scatterplot in Fig. 1A after standardization. 
Standardization of neuronal discharge rates (by sub- 
tracting the mean and dividing by the S.D.) normalized 
the activity rates of all neurons to 0.0, and their vari- 
ances to 1.0. Even though PCA is a distribution free 
method for transformation of multivariate data, infer- 
ences about the statistical significance of eigenvalues 
and eigenvector coefficients are based on normally dis- 
tributed data. Though we have found that integrated 
spike-train data tend to approximate Poisson or super- 
Poisson distributions, they tend to converge on a nor- 
mal (Gaussian) distribution as they are integrated over 
longer time periods (demonstrated in Fig. 1C) or larger 
populations. Thus, the issue of data distribution can be 
handled in several ways. First, large numbers of data 
samples are used. Most parametric statistical techniques 
are quite robust against distribution anomalies when n 
is large (e.g. over 100). Second, Poisson distributed 
integrated spike train data converge on a normal distri- 
bution when larger integration times are used, or when 
large numbers of neurons are integrated into an eigen- 
function. Finally, one can utilize square-root transfor- 
mations to force Poisson distributed data into a more 
normal distribution. We have used such transforma- 
tions routinely but have not found the results to be 
substantially different from analyses using untrans- 
formed data. 


3.4. Calculation of principal components 


Fig. 1D illustrates the use of PCA to rotate the 
scatterplot in Fig. 1B, which itself was constructed by 
standardizing the data in Fig. 1A. The rotation of the 
3D scatterplot was produced by using PCA-derived 
coefficients for each component (PCI: 0.707 for both 
neurons; PC2: 0.707 and — 0.707 for neurons 2 and 7, 
respectively). The set of coefficients for each PC defines 
a rotation vector (eigenvector) which, by dot product 
multiplication of the coordinates of the points in Fig. 
1B, produced the scatterplot in Fig. 1D. This counter- 
clockwise rotation produced a new X-axis (now called 
PC1) which is aligned along the long axis of the scatter- 
plot. The position of a point along this new axis now 
constitutes an optimal measure of whatever factor(s) 
were originally responsible for producing the correlated 
discharge between the neurons. 

The eigenvalues, representing the percentage of the 
total variance explained by each of these principal 
components were calculated as 70.13 for PCl, and 
29.87 for PC2. PC1 can be considered as a mathemati- 
cal definition of a new coordinate system which paral- 
lels the major factor of interaction between these 
neurons, defined purely in terms of their correlation in 
this data set. PC2, which must be orthogonal to compo- 
nent 1, accounts for the remainder of the variance, 
which may represent noise, or a less significant factor of 
interaction between the variables. 


3.5. Using PCA to map functional relationships within 
neuronal ensembles 


Distinct, functionally significant patterns of PC 
weightings emerged when the above techniques were 
applied to data from larger neuronal ensembles. Fig. 2 
illustrates the PC weightings obtained through analysis 
of the 23-neuron data set discussed above. To investi- 
gate the relationship between these weightings and the 
topographic representation of whiskers in the VPM, 
each neuron was graphically rank ordered according to 
the rostrocaudal position of its RF center, i.e. its princi- 
pal whisker, an accurate indicator of its actual position 
in the VPM. This provided visualization of the spatial 
attributes of the PC weighting patterns, allowing them 
to be interpreted rather like a receptive field (RF). 

Fig. 2A provides a good example of the multi-topo- 
graphical patterns found in PC weightings throughout 
this study. PCl (top) invariably contained relatively 
homogeneous, all positive weightings, and was there- 
fore essentially non-topographical. As such, PC1 pri- 
marily encoded the magnitude of global activity in this 
ensemble: the neurons most weakly weighted in this PC 
were found to be those which had very weak responses 
to sensory stimuli. Neurons 21—23 (from left) exhibited 
no clear RFs, and neurons 6 and 8-10 exhibited very 
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Fig. 2. Weighting coefficients of principal components derived from 
the same set of 23 neurons are similar over different experimental 
paradigms. (A) Linear graphs depicting weighting coefficients of 
principal components (PCs) 1-6 calculated from simultaneous 
recordings of 23 neurons in the VPM thalamus during an 1821.49 s 
sequence of spontaneous behavior in an awake rat. The 23 neurons 
are rank ordered on the X-axis according to the rostrocaudal loca- 
tions of the centers of their RFs on the mystacial whiskers. The 
RF centers were quantitatively determined using peri-whisker-stimu- 
lation histograms. The mystacial whiskers in rat are commonly 
divided into rows A—E (dorsal-ventral), and columns 1-7 (caudal- 
rostral). The much smaller guard hairs (GH) are rostral to the 
mystacial whiskers. When more than one neuron was centered on 
the same whisker (e.g. the three neurons on D6) they were rank 
ordered according to the value of their coefficients in PC2. Three 
neurons did not have RFs on the whiskers (labeled ‘N’). (B) Simi- 
lar linear graphs depicting principal components derived from 
recordings of the same neurons in an experiment (20 min after that 
in (A)). For most of this experiment the animal remained still, 
allowing whisker E2 to be deflected using a computer controlled 
hand-held vibromechanical stimulator. The X-axis contains the 
same rank ordering of neurons as in (A). Note that the PCs are 
similar to those in (A), but are reversed in polarity and changed in 
order (e.g. PC3 and PC4 here are reversed in polarity and order 
compared to (A)). 


weak sensory responses even though they had RFs 
centered on whisker D4. In contrast, PC2 almost lin- 
early encoded the neurons according to the rostrocau- 
dal position of their principal whiskers on the face. 
(The D6 whisker was weighted most negatively; the E2 
whisker most positively.) To a lesser extent, PC3 en- 
coded dorsoventral position, positively weighting the 
two neurons with RFs in the caudal C-row (both 
covering C2, but also extending to the D-row), and 
negatively weighting the two neurons with RFs in the E 
row. 

Regression analysis was used to statistically validate 
this observation that PCs 2 and 3 defined gradients 
across the whisker pad. The weights for PCs 1-4 
(dependent variables) were regressed against the spatial 
positions of the neurons’ RF centers (independent vari- 
ables). These RF centers were defined either in vertical 
coordinates (whisker rows A— E, numbered 1-5) or 
horizontal coordinates (whisker columns 1-6, and 
guardhairs = 7). When the RF centers covered multiple 
whiskers, the midpoint between them was used. The 
three neurons without RFs were omitted. When PCs 
1—4 were regressed against the neurons’ column num- 
ber, PC2 yielded a coefficient of determination (R7) of 
0.69 (F=41.4; P=5x10~°), while the R’s of the 
other PCs were insignificant (PC1: 0.006, PC3: 0.009, 
PC4: 0.0003). Thus, the rostrocaudal positions of center 
RFs on the whisker pad were robustly and selectively 
encoded by PC2. Similarly, dorsoventral position was 
selectively encoded by PC3, even though this sample 
contained relatively few cells with RFs in the C and E 
rows: When regressed against row number, PC3 yielded 
an R? of 0.51 (P = 0.0003), while the R’s of the other 
PCs were insignificant (PC1: 0.02, PC2: 0.01, PC4: 
0.008). As described below (Fig. 3), such results were 
quite consistent across the eight animals used for this 
study. 

Higher numbered components encoded information 
on statistical outliers which here tend to reflect higher 
spatial frequencies. For example, PC4 differentiated 
between neurons with RFs in the caudal D row vs E 
row. PC6 differentiated between neurons with RFs in 
the rostral whisker pad. Finally, PCS positively 
weighted the neurons without RFs. PCS also negatively 
weighted neurons with RFs centered on whiskers E2 
and D6. This seemingly unlikely combination is remi- 
niscent of our previously reported finding (Nicolelis et 
al., 1993, 1995) that many neurons which respond at 
short latency (4-10 ms) to stimulation of caudal 
whiskers (e.g. E2) often respond at longer latency (15— 
25 ms) to stimulation of the rostral-most whiskers (e.g. 
D6). Thus, the higher numbered principal components 
have successively more complex weighting patterns, 
whose functional significance is less clearly related to 
absolute RF position. 
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Fig. 3. Weighting coefficients of principal components derived from 
VPM neuronal ensembles recorded in additional animals. All were 
constructed using the same technique as in Fig. 5: Correlation 
matrices were constructed from neural activity measured during all 
10-ms time intervals in long periods of spontaneous exploratory 
whisking behavior. RF locations were determined from quantitative 
RF mapping experiments carried out just after the spontaneous 
behaviors. Graphs depict the weighting coefficients in PC1—3 (‘PC 
WEIGHT’ on Y-axis) for each of the neurons, which are rank 
ordered along the X-axis according to rostrocaudal location of their 
RF centers ((WHISKER’). (A) Three line plots depict weighting 
coefficients of PC1—3, obtained from a PCA carried out on data from 
21 VPM neurons simultaneously recorded during an 1545.43 s se- 
quence of spontaneous exploratory whisking behavior in an awake 
rat. The RF centers of these 21 neurons ranged from the guard hairs 
just rostral to rows E and D (‘GhE’ and ‘GhD’) to the caudal large 
whiskers in the C, D, and E rows. When these were rank ordered 
according to rostrocaudal position, the PC2 weights reveal a roughly 
linear progression wherein the neurons with rostral RFs (GhE) are 
most negatively weighted, while those with caudal RFs (E2) are most 
positively weighted. PC3 weights neurons according to the dorsoven- 
tral position of their RFs: those with dorsal RFs (on the Cl and C2 
whiskers) are most negatively weighted, while those with ventral RFs 
(E-row) are most positively weighted. (B) Similar line plots depict 
PCs 1-3 derived from eight VPM neurons in another rat, simulta- 
neously recorded during 1812.965 s of spontaneous exploratory 
whisking behavior. The neuronal weightings in PC2 defined a roughly 
rostrocaudal gradient, most positively weighting the guard hairs 
rostral to the A-row (GhA), and most negatively weighting the B4 
whisker. PC3 roughly defined a dorsoventral gradient, with most 
negative weightings on neurons with RFs on the guard hairs rostral 
to the C-row, and neutral weightings on neurons with larger fields 
which cover both GhC and GhA (GhC-A). 


3.6. PCA results are similar even when derived in 
different conditions 


Additional evidence for the high statistical reliability 
of these components is provided by the observation 
that the components changed only slightly when calcu- 
lated from repeated experiments on the same neuronal 
ensembles. As is illustrated in Fig. 2A,B, this consis- 
tency in PCs was observed even when different time 
bins were used (10 ms in Fig. 2B vs 25 ms in Fig. 2A). 
In Fig. 2B the neuronal weights are rank ordered 
exactly as in Fig. 2A, revealing remarkably similar 
overall weighting patterns: PC2 is virtually the same in 
Fig. 2A,B, except that its polarity is reversed (insignifi- 
cant in PCA). PC3 in Fig. 2A is equivalent to PC4 in 
Fig. 2B, although their polarities are reversed. This 
reversal of PCs 3 and 4 suggests a relatively slow time 
course of interactions between neurons with dorsal vs 
ventrally located RFs. 

The weightings in Fig. 2C were remarkably similar to 
those in Fig. 2B, even though the PCs in Fig. 2C were 
derived from recordings of the same neurons obtained 
40 min later during an experiment in which the E2 
whisker was repetitively stimulated during periods of 
behavioral immobility (comprising about 80% of total 
experimental time). The only major difference between 
PCs 1-3 in these experiments is that PC3 in Fig. 2C 
reveals a relatively greater weighting of whisker E2, 
which was selectively stimulated during this experiment. 
The higher numbered components also exhibited re- 
markable equivalences, though altered in polarity and 
eigenvalue: PCS in Fig. 2B,C is similar in weighting 
pattern, though reversed in polarity. Moreover, both 
are similar to PC6 in Fig. 2A. Finally, PC6 in Fig. 2A 
and PCS in Fig. 2B,C contain similar weightings of the 
neurons on the left side of the figure (i.e. C5—D6 
through D4). 

To conclude, the basic weighting structure of the PCs 
derived from this neuronal ensemble remained remark- 
ably constant over experimental time and over changes 
in bin size and behavior (e.g. spontaneous active whisk- 
ing vs passive whisker stimulation). The proportions of 
time spent during these different behaviors were 
reflected more in the PC eigenvalues (and therefore the 
ordering of PCs) than in the PC weightings. Further- 
more, the remarkable similarities between the compo- 
nents in Fig. 2B,C (which both use 10-ms_ bins) 
demonstrates a high level of statistical reliability of 
these calculations, even when relatively small data sets 
were used (Fig. 2C was derived using a 550 s experi- 
ment, vs 1821 s in Fig. 2A,B). 


3.7. Statistical significance of PCs 


The statistical significance of PCs can be assessed 
through calculation of S.E. of the eigenvalues and 
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weights (see Section 2). For the components in Fig. 2A, 
the average weighting coefficient in PC1 was calculated 
to be 12.98 times its S.E., and for the combined PCs 1—6 
was 13.44 times the S.E. Moreover, only weights which 
were very close to zero were less than three times their 
S.E. Thus, the weights with low statistical significance 
had negligible impact on the components overall. 

The S.E. of the eigenvalues, which specify the variance 
associated with each component, also tended to be very 
small fractions of their values. For example, the eigen- 
value of PC1 in Fig. 2A was 4.26 + 0.022 S.E., meaning 
that it explained a variance equivalent to 4.26 of the 
original neurons (18.5% of the total; each neuron ex- 
plains a standardized variance of 1.0). Though for 
typical implementations of PCA this is not a particularly 
high eigenvalue, it is quite remarkable considering the 
complexity of the information carried within the dis- 
charge of 23 highly stochastic neurons during free 
behavior. We observed that PC1’s eigenvalue increased 
to 14.56 when the time integral was raised to 500 ms, 
suggesting that much of the complex information and/or 
noise in this ensemble was expressed over shorter time 
periods. 


3.8. Comparability of PCA results across animals 


Similar results were obtained when PCA was used to 
analyze data from VPM neuronal ensembles in the seven 
other animals, two of which are shown in Fig. 3A,B. 
Though the neurons recorded in each animal had differ- 
ent RF mappings and discharge characteristics, and a 
range of time integrals (S—25 ms) were used, they tended 
to produce similar weighting patterns in their PCs. In all 
animals, PCl contained roughly homogeneous positive 
neuronal weightings, and PC2 and, to a lesser extent, 
PC3 roughly encoded the position of RF centers accord- 
ing to particular gradients across the whisker pad. In 
Fig. 3A, for example, PC1 has relatively homogeneous 
weightings, but PC2 differentiates neurons with far 
rostral vs far caudal RFs. In contrast, PC3 sharply 
discriminates between neurons with RFs on row C vs 
row E. Using the regression analysis described above for 
Fig. 3A, the PC2s of five or the total eight animals were 
found to significantly (P<€0.01) encode rostrocaudal 
position. In four of eight of these animals the PC3s also 
significantly encoded dorsoventral position of neuronal 
RFs. Furthermore, the PCs in Fig. 3B showed a variant 
of this relationship: PC2 most selectively encoded the 
dorsoventral position (R* = 0.60, P = 0.02), while both 
PC1 (R?=0.70) and PC2 (R* = 0.64) encoded rostro- 
caudal position (P = 0.01 for both). In none of the other 
seven animals did PC1, PC4 or PCS yield any significant 
encodings of overall rostrocaudal or dorsoventral gradi- 
ents. These statistics, therefore, support the general 
conclusions that: (1) PC1 generally contains roughly 
homogeneous positive neuronal weightings, (2) PC2 


generally encodes the rostrocaudal position of RF cen- 
ters across the whisker pad, (3) PC3 may encode overall 
dorsoventral position, (4) higher numbered PCs tend to 
encode more fine-grain spatial relationships, often defin- 
ing sharp boundaries between adjacent whiskers or 
whisker groups. 

These overall results were obtained despite the wide 
range in ensemble size and RF distribution. For exam- 
ple, the ensemble used for the PCs in Fig. 3A included 
21 neurons, and that for Fig. 6B had eight neurons. 
Whereas the RFs in Fig. 3A covered the whole rostro- 
caudal extent of the whisker field, those in Fig. 3B 
covered a more limited area, mainly including the rostral 
whiskers and guard hairs. As such, PC2 and PC3 in Fig. 
3B defined dorsoventral and rostrocaudal gradients 
across the rostral face, rather than across the whole 
whisker field. Thus, despite the obvious disadvantages 
associated with use of relatively small inhomogeneous 
data sets (as in Fig. 3B), the present results reveal a 
remarkably robust general tendency for successive PCs 
to encode sensory information in the form of succes- 
sively finer spatial resolutions. 

This correlation of PCs with spatial frequency bands 
corresponds closely to the components which we have 
derived using data from computer simulated sensory 
systems (unpublished observations) in which a model 
receptor sheet was activated with moving stimuli. This is 
remarkable considering that the PCs here were derived 
from VPM neuronal ensembles which received no sen- 
sory stimulation other than that produced by the animal 
itself during the normal course of spontaneous behavior. 


3.9. Single neurons contribute variance to multiple 
coding dimensions 


By definition PCs, such as those shown in Figs. 2 and 
3, are othogonal and therefore define mappings in 
different dimensions. The fact that at least the first 3-6 
PCs were found to exhibit distinct and consistent func- 
tional topographies suggests that these PCs may describe 
real dimensions of information processing. If so, each 
neuron may be considered to contribute a portion of its 
variance to several of these dimensions, as statistically 
measured by the square of its PC weighting (see Section 
2). These weights, therefore, offer methods for defining 
each neuron’s role in multiple dimensions of processing, 
and for mapping the variance weighted configuration of 
the neuronal ensemble involved in each of these func- 
tional tasks. 

Fig. 4, for example, depicts the position of each 
neuron in Fig. 2A as a point in a space defined by PC1 
and PC2 (Fig. 4A) and by PC3 and PC4 (Fig. 4B). 
Each neuron is labeled according to its principal 
whisker, as depicted in Fig. 2A. For illustration, con- 
sider the three neuron pairs whose principal whiskers 
are C2, D2 and E2. In PCs | and 2 (Fig. 4A) they are 
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Fig. 4. Use of PCA to define functional relationships between neu- 
rons. (A) Neurons 1—23 (from Fig. 2A) plotted as dots according to 
their weightings in PCl and PC2. (B) Neurons 1-23 plotted accord- 
ing to their weightings in PC3 and PC4. 


Fig. 5. Plotting neuronal population responses to whisker stimulation 
in a PCA-defined space reveals spatial arrangement of whiskers. 
Neuronal population responses to deflection of each of nine different 
stimulated whiskers were plotted within a space defined by PCs 2 and 
3 (each dot representing the averaged responses to 300-600 deflec- 
tions of the indicated whisker). The responses were measured during 
the peak short latency response period, i.e. between 5 and 8 ms 
post-stimulus (as shown in B and C). The arrangement of dots on this 
plot is generally consistent with the spatial arrangement of the 
indicated whiskers on the face. 


part of two continuums, hypothesized to encode magni- 
tude and rostrocaudal position, respectively. Each of 
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these six neurons contributes 3—14% of the total vari- 
ance to each of these encodings. In PCs 3 and 4 (Fig. 
4B) they play a very different role, defining a 2D space 
in which objective spatial position appears to be much 
less important than high-contrast differentiation be- 
tween the three whiskers (C2, D2 and E2). Each of the 
six neurons contributes a high percentage of its vari- 
ance to this task (up to 36% for C2’s weighting of PC3). 
Similarly selective differentiations of caudal whiskers 
were also observed in other animals used in this study. 
This selectivity may be explained by the fact that rats 
tend to employ their relatively short rostral whiskers as 
a group to touch objects during exploration, while their 
caudal whiskers project at widely different angles into 
space. 

The present results suggest that the optimal mapping 
of afferent information by PCA (and the brain itself) 
may extract multiple different significant features of the 
sensory experience, using relatively small portions of 
each neuron’s variance to generalize to global objective 
mappings of space or magnitude. Meanwhile, more 
local and selective dimensions of information process- 
ing consume larger amounts of variance from small 
subsets of neurons. 


3.10. PCs classify neuronal population responses to 
stimulation of specific whiskers 


The PCs derived here were found to be very effective 
tools for classifying sensory responses. In particular, 
PC2 and PC3 provided a highly efficient 2D subspace 
for clustering the neuronal population responses to 
discrete stimulation of different whiskers. To demon- 
strate, Fig. 5 shows a ‘mapping’ of each of the nine 
whiskers stimulated in this experiment into a space 
defined by PCs 2 and 3. The x or y coordinates of each 
point on this graph were defined by calculating eigen- 
functions (see Section 2) for both PC2 and PC3 for 
stimulation of each of these nine whiskers. Here, eigen- 
functions 2 and 3 were each calculated as the sums, 
weighted respectively by PCs 2 and 3, of the spiking 
responses (in the 4- to 8-ms latency epoch) of all 23 
neurons to 300-600 stimulations of the indicated 
whisker. The resulting scatterplot shows existence of a 
unique, anatomically appropriate position for each 
whisker, especially in the rostrocaudal dimension. 
Moreover, it accurately plotted the positions of 
whiskers which were not within the center-RFs of any 
neurons within the sample, but were within the sur- 
round-RFs of some neurons (e.g. whiskers C4—6 and 
B3). These results therefore show that PCs derived from 
activity of a relatively small and biased sample of VPM 
neurons can effectively represent the spatial positions of 
a range of whisker stimulus positions. It is important to 
note, however, that this map does not reproduce the 
absolute X—Y positioning of the whiskers on the face. 
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Instead it provides a functional mapping of these 
whiskers, as defined by the response patterns of this set 
of neurons in the VPM occurring during spontaneous 
behavior. 

Roughly similar whisker mappings were found in the 
eigenfunction 2—3 data from six of the other seven 
animals. Though these mappings were somewhat biased 
by the particular selection of neurons in each data base, 
the caudo-ventral whiskers tended to occupy larger 
areas of these maps than the dorsal or rostral whiskers. 
This is, of course, consistent with the anatomical map- 
ping of these whiskers in the somatosensory system 
(Chapin and Lin, 1984). To conclude, this demonstra- 
tion argues that population covariance patterns, even 
that occurring during general spontaneous behavior, 
contain a significant amount of information on the 
mapping of whisker location within a generalized 2D 
space. This space, however, is not an accurate objective 
map of the whiskers, or even their receptor densities, 
but is warped according to the patterns of temporal 
correlation between different whisker inputs. Finally, 
the proportion of population variance dedicated to this 
mapping is relatively small, in this case about 12% of 
the total (as calculated from eigenvalues in Fig. 2A). 
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Fig. 6. Higher numbered PCs encode higher frequency domains of 
neuronal population responses to moving whisker stimuli. Peri-stimu- 
lus responses of PCs 1—4 to moving a probe across the mystacial 
whiskers. Each trace is the average of the indicated PCs over 17 trials. 
Initial contact of the probe on the caudal whiskers is indicated by 
vertical dotted lines. Brushing consisted of slowly moving a hand held 
cotton probe tip across the whiskers in alternate directions along an 
oblique caudo-dorsal to rostro-ventral axis. The bar above the traces 
indicates the approximate timing of the probe sweep in the rostro- 
ventral direction. Vertical axes depict average equivalent discharge 
rates of eigenfunctions: small ticks, 10 Hz. Horizontal axis depicts 
time (s) before and after the stimulus onset. See text for explanation 
of PC responses. 
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3.11, Feature detection by principal components 


The above results also showed that the PCs may be 
used, in conjunction with data from simultaneously 
recorded neuronal populations, to construct feature 
detectors (i.e. eigenfunctions) whose general functional 
effectiveness far surpasses data measured from any 
single neuron. The following figures demonstrate that 
the eigenfunctions derived from the PCs in Fig. 2A are 
useful for detection of important events during ongoing 
behavior, not only when applied to multi-trial averaged 
data, but also to single trial (continuous) data. 

The first question was whether eigenfunctions can be 
used to estimate the magnitude, velocity or direction of 
stimulus movement across the whiskers. Temporal pat- 
terns of activity of the eigenfunctions were observed to 
depict important features of stimulus movement, even 
though this study used only spatially related variables. 
(Neuronal activity was correlated only within the same 
time bins.) For example, the peri-event histograms in 
Fig. 6 show the averaged temporal responses of eigen- 
functions 1—4 (from the PCs in Fig. 2A) to 17 brush- 
ings of a bar in a dorsocaudal-to-ventrorostral direction 
across the whiskers in an awake, immobile rat. The 
probe’s successive contact with different whisker groups 
(as observed in frame-by-frame video analysis) is appar- 
ent in the responses of the various eigenfunctions: the 
caudal C-row whiskers were touched first, which pro- 
duced peak ‘A’ in eigenfunction 3. (PC3 has highly 
positive weightings for neurons with RFs centered on 
whisker C2.) The probe then touched the caudal D-row 
whiskers, producing peak ‘B’ in eigenfunction 4. (PC4 
positively weights the caudal D-row.) The probe then 
touched the caudal E-row whiskers, which through 
their negative weightings in PC3 and PC4, produced the 
valleys after ‘A’ in eigenfunction 3, and at ‘C’ in 
eigenfunction 4. The peaks following the valleys in 
eigenfunction 3 and eigenfunction 4 (around ‘D’) were 
mainly caused by ‘bouncing back’ of the caudal 
whiskers previously bent forward by the probe. Over 
the same time period (from ‘A’ to ‘C’) eigenfunction 2 
exhibits a broader peak which declines slowly toward a 
deep valley at ‘E’. This is explained by the fact that PC2 
weights the caudal whiskers positively and the rostral 
whiskers negatively. Finally, eigenfunction 1, which 
positively weights all neurons, regardless of their 
whisker RFs, exhibits a peak which covers, and reflects 
the overall intensity of, the entire probing movement. 

The above results demonstrate how the eigenfunc- 
tions can be used to process time-varying sensory input 
information by dividing it into ever finer spatiotempo- 
ral frequency domains. When a sensory surface inter- 
acts with a moving stimulus, its spatial frequencies are 
revealed as temporal frequencies. Here, successive PCs 
were associated with decreasing peak-to-valley dura- 
tions of their sensory responses to the moving stimulus: 
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Fig. 7. PC2 selectively detects tactile contact on rostral whiskers. Peri-event averages of PCs 1—4 centered around the onset of 23 time periods 
when the rat contacted objects with the rostral mystacial whiskers. All these behavioral events were detected using frame-by-frame analysis of 
videotapes which were synchronized with data collection during the experiment. This further demonstrates PC2 codes (with negative polarity) for 
tactile exploration with the rostral whiskers. XY-axes: pre- and post-touch time (s). Bins, 4 ms; small ticks: 20 ms. Y-axes: average equivalent firing 


rate, as in fig. 11. Small ticks: 1.0 spike/s. 


~ 300 ms for PCl, ~ 150 ms for PC2, ~70 ms for 
PC3, and ~35 ms for PC4. This combination of 
different spatial frequencies should optimally describe 
the linearizable information carried as covarying activ- 
ity in this recorded neuronal population. 


3.12. Principal component eigenfunctions resolve 
spatiotemporal patterns of whisker contact during 
exploratory whisking 


Normally, rats obtain sensory information from their 
vibrissae through active whisking behavior. One of the 
critical applications of the eigenfunctions derived here 
was for the detection of particular active whisker move- 
ments across objects in the environment. This allowed a 
comparison of the sensory information obtained during 
whisker exploration, as opposed to passive stimulation 
with a probe. 

In all eight animals used here, PCs 1—4 were consis- 
tently found to detect significant features of tactile 
contact on objects during tactile whisking. To demon- 
strate the typical robustness of these responses, Fig. 7 
shows averages of the same eigenfunctions 1—4 cen- 
tered around 23 instances of active exploratory contact 
of the rostral whiskers against objects (times deter- 
mined by frame-by-frame video analysis). Though ei- 


genfunction 2 exhibits a consistent and _ highly 
significant negative response during such behaviors, the 
responses of eigenfunctions 3 and 4, which mainly 
encode the caudal whiskers, show no significant re- 
sponses. Finally, eigenfunction 1 exhibits a response 
pattern which is consistent with the generally increased 
whisker contact surrounding these behaviors. Similar 
peri-event averages of eigenfunctions were used to ver- 
ify that the behavioral correlates of the above PCs 
depicted in Fig. 7, and equivalent analyses of data from 
the other seven animals, were consistent throughout the 
entire duration of the experiment. 

Eigenfunctions, especially those constructed using 
populations of 20 or more neurons, were easily capable 
of detecting significant behavioral events on a single- 
trial, real-time basis. For illustration, Fig. 8 shows the 
same eigenfunctions 1—4 over a 20-s period of sponta- 
neous behavior. Video analysis showed that this partic- 
ular sequence was of interest because it consisted of a 
series of active head movements which brought the 
mystacial whiskers into contact with a Plexiglas wall 
just to the right of the animal. This allowed an analysis 
of the responses of the 23 neurons in the VPM to active 
movement of the whiskers in various directions across a 
smooth, flat tactile object. Particular tactile movements 
observed in frame-by-frame video analyses were repre- 
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Fig. 8. Population vectors define timing and direction of tactile whisker contact. (A) Rasters of spiking activity of 23 simultaneously recorded 
neurons over a 1.5-s period. (B) Stripcharts of continuous eigenfunctions of PCs 1—4 over the same period during which the rat moved its whiskers 
in various directions against a wall. Letters A—G represent different movements of mystacial whiskers against a wall (see text), as observed in 
frame-by-frame analysis of videotape records synchronized to data collection in this experiment. (C) Activity of the same PC’s over a 20-s period. 
Sequence B is contained between the vertical lines. Y-Axis shows the absolute time (s) in the experiment for all recordings: Bins, 40 ms; small ticks, 
200 ms. Y-Axes show equivalent ‘firing rates’ of the standardized weight-summed eigenfunctions: small ticks, 10 spikes/s. Horizontal lines through 
the eigenfunctions indicate 2.0 S.D. away from the mean spontaneous activity, as calculated during quiet resting behavior (each S.D. is 13.2 S.E.). 
Dotted line under A is 12 S.D.s above the mean; dotted lines under C and E are six S.D.s above the mean. Eigenfunctions were calculated as in 
Section 2 using 10-ms time integrals, and then smoothed into bins by averaging within a three-bin moving window. 
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sented in the eigenfunctions as peaks or valleys which 
were clearly distinguished from their ‘background’ ac- 
tivity. In Fig. 8B, C, the statistical significance of these 
peaks and valleys can be discerned by their relation to 
the confidence limits shown as horizontal dotted lines 
on each trace, defining eigenfunction activity levels two 
standard deviations (S.D.s) in either direction from the 
means, as measured during periods of pure resting 
behavior in the same experiment. 

The sequence in Fig. 8B, as demarcated by vertical 
dotted lines in Fig. 8C involved the following specific 
movements of whiskers against the tactile surface, asso- 
ciated with peaks and valleys in various eigenfunctions: 
The peak at ‘A’ in PC] represents the relatively nonse- 
lective global ensemble response to a sequence of active 
head and whisker movements commencing with a sud- 
den head movement to the right and then down, sweep- 
ing the whiskers first in a backward and then 
downward direction against the wall. Statistically, this 
sequence in the eigenfunction 1 is highly significant, 
reaching up to 12 S.D.s from mean. 

The higher numbered eigenfunctions also exhibit sig- 
nificant peaks which are more selective for particular 
parts of this behavioral sequence. For example, at the 
onset of rightward head turning eigenfunction 4 (PC4) 
exhibits a highly significant (10 S.D.s from the mean) 
negative deflection (‘B’ in Fig. 8B, C). This was pro- 
duced by the downward and backward movement of 
the ventral whiskers against the wall, preferentially 
stimulating the caudal E-row whiskers. This negative 
deflection was predictable based on the pattern of 
weighting coefficients for PC4, which were highly nega- 
tive for the two neurons with RFs on whisker E2. This 
response, however, was not simply formed by the spuri- 
ous spiking of two single neurons. PC4 also negatively 
weighted two neurons with RFs on D4 and three 
neurons with no RFs. Examination of the raw spiking 
data for the total seven neurons with negative weight- 
ings for PC4 showed that five exhibited increased spik- 
ing during the negative deflection shown at B. 
Furthermore, the five neurons with strongly positive 
weightings in PC4 (>0.1) all exhibited decreased dis- 
charge during this same period. Thus, even this rela- 
tively small deflection of an eigenfunction for a high 
numbered component represents the highly significant 
joint activity of a neuronal subpopulation. Similar 
whisker movements against the wall were associated 
with other negative deflections of eigenfunction 4, such 
as at D and J in Fig. 8B, C. 

PC2, which negatively weights rostral whiskers and 
positively weights caudal whiskers, was active during 
normal exploratory whisker movements in which the 
whiskers moved across objects in a rostral-to-caudal 
direction. An example is seen in PC2 in Fig. 8B, C in 
which the rostral-to-caudal succession of whisker con- 
tacts is reflected as negative-to-positive progression 


shown under ‘C’. In other cases, such as in ‘K’, the 
rostral whiskers alone were used to explore objects on 
the floor, producing a pronounced depression in eigen- 
function 2. Positive deflections of the eigenfunction 2 
also carry important information consistent with its 
positive weighting of the caudal whiskers: the peak at 
‘H’ in Fig. 8C depicts a right turn of the head which 
moved the caudal whiskers in a backward direction 
against the wall. Overall, these results demonstrate that 
PC2 encodes the direction of whisker sweeping across 
objects in the rostrocaudal axis. Similarly, eigenfunc- 
tion 3, whose PC(3) positively weights neurons with 
RFs in the more dorsal whiskers (rows B and C), was 
most active during movements in the dorsoventral di- 
rection. As an example, ‘E’ shows the response of 
eigenfunction 3 to dorsalward movements of the 
whiskers against the wall. Similar, but shorter move- 
ments occurred at ‘T’, ‘L’ and ‘M’. 


3.13. Eigenfunctions reveal global and local neural 
ensemble information in continuous time 


Eigenfunctions were also employed here to depict 
neural ensemble information in stripcharts, which al- 
lowed visualization of the state of multiple eigenfunc- 
tions in continuous time. These depicted the 
moment-to-moment status of multiple dimensions of 
population information which could never be observed 
in analyses at the single neuron level. For example, the 
eigenfunction stripcharts in Fig. 8 reveal the presence 
and time course of distinct spontaneous oscillatory 
phenomena. The onset and offset timing of these 
rhythms could not be detected in peri-event averages, 
but instead required continuous-time depiction of the 
state of the neuronal ensemble. We have observed these 
8-12 Hz oscillations in the PCls of neuronal ensembles 
in the somatosensory thalamus or cortex in all of the 20 
animals recorded for this study, and have routinely 
utilized this ability of eigenfunction 1 to depict such 
oscillatory phenomena in real time and to carry out 
precise analyses of their functional characteristics 
(Nicolelis et al., 1995). Video analyses (see Section 2) 
showed that these oscillations characteristically began 
during the period of attentive immobility which just 
precedes active whisker twitching, and continue until 
onset of larger active exploratory movement. They ap- 
pear, therefore, to represent a global neurophysiological 
process which pervades the sensorimotor thalamocorti- 
cal system during behavioral preparation for movement. 

Whereas eigenfunction 1 typically reveals major 
global phenomena such as spontaneous global oscilla- 
tions, the higher numbered components typically code 
for more ‘local’ information. To illustrate further, Fig. 
9 shows a raster (A) of the simultaneous activity of these 
23 cells plus PCs 1 and 2 (B) over a 1.4-s period during 
which the E2 whisker was subject to a single (3°; 
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at ‘S’) mechanical displacement. While eigenfunction 1 
again clearly reveals the global 8-12 Hz oscillations, 
the higher numbered eigenfunctions follow them only 
weakly. Instead, eigenfunctions 2—4 exhibit selective 
responses to the E2 whisker stimulation (S) which are 
clearly differentiated from the oscillatory peak (O) that 
appears slightly later. 

The mechanism by which the eigenfunctions can 
selectively filter different dimensions of global and local 
information in neuronal ensembles involves the use of 
positive and negative neuronal weightings to selectively 
‘subtract’ the information already encoded by lower 
numbered components. Since PC1 normally has a rela- 
tively homogeneous set of exclusively positive weight- 
ings, it identifies ‘global’ functions, such as the 8—12 Hz 
oscillations, which are characteristic of nearly all neu- 
rons in the ensemble. These typically constitute the 
greatest source of variance in eigenfunction 1, but are 
effectively cancelled out in the higher numbered compo- 
nents. Thus, the orthogonalization of network covari- 
ance by PCA effectively differentiates between global 
and local sources of neural information. 

At time ‘S’ in Fig. 9B, local information is presented 
in the form of a sensory stimulus which differentially 
affects the neurons in the ensemble. The responses of 
the different eigenfunctions to this stimulus can be 
predicted by their weightings of neurons with RFs 
centered on the stimulated whisker (E2) and its neigh- 
bors (e.g. D2, D4 and C2) all of which respond at some 
level to E2 stimulation. Whereas PC2 weights E2 and 
these neighbors negatively, PC3 and PC4 selectively 
weight E2 positively, but some of its neighbors nega- 
tively. Because of the fact that whisker E2 was stimu- 
lated alone, and not in conjunction with its neighbors, 
eigenfunction 2 produced a cleaner resolution of the 
stimulus than did eigenfunction 3 or eigenfunction 4. 
Finally, the eigenfunction for PCS, which has near zero 
weightings for all neurons with RFs around whisker 
E2, exhibited no response to this stimulus. 

The response selectivity of eigenfunctions 2—4 to 
whisker E2 stimulation was verified in peri-stimulus 
averages (Fig. 9C). Whereas the whisker stimulation 
increased the magnitude of the average eigenfunction 1 
equivalent ‘discharge rate’ from a mean 8.2 to maxi- 
mum 15.7 Hz after the stimulus, for a signal/back- 
ground (S/B) ratio of 1.93, the activity of PC2 increased 
(with negative polarity) from a mean — 0.3 to — 5.9 Hz 
after the stimulus, for a S/B ratio of 18.9. Even when 
the high background activity of eigenfunction 1 is 
disregarded by calculating the responses in terms of 
deviation from the mean, its maximal response to 
whisker E2 stimulation was only 15.2 S.D.s from the 
pre-stimulus mean, compared with 24.0 S.D.s for PC2, 
14.5 S.D.s for PC3 and 22.9 S.D.s for PC4. This greater 
resolution of E2 whisker stimulation by PC2 is more 
remarkable when one considers that the higher num- 


bered components by rule explain much less total statis- 
tical variance than PC1. Here, the eigenvalue for PCl 
was 3.8, and for PC2, 1.5. Thus, PC2 devoted a much 
higher proportion of its total variance to E2 whisker 
responses than does PC1. 


4. Discussion 
4.1. PCs and multidimensionality 


In this study, eigenvalue decomposition of neuronal 
ensemble activity in the VPM measured during sponta- 
neous behavior yielded at least 3-6 uncorrelated fac- 
tors (PCs) whose resolution of significant information 
in the ensemble (i.e. explained variance) was consider- 
ably greater than that of any single neuron. This finding 
appears to validate our hope that PCA could success- 
fully provide an optimal linear representation of the 
‘signals’ in these recordings, which are concentrated in 
the first few components, while filtering out ‘noise’ 
(mainly uncorrelated neuronal activity), which is se- 
questered in the remaining components. Because of the 
orthogonality criterion for PCs, each represented a 
separate ‘dimension’ of covariant activity in the ensem- 
ble. Moreover, each of these PCs was found to possess 
a clear and distinct functional attribute, suggesting that 
together they provide clues toward resolution of the 
fundamental axes around which the different dimen- 
sions of information processing might be carried out 
within the thalamocortical circuitry. These findings are 
consistent with the general framework of parallel dis- 
tributed processing in neuronal networks (Rumelhart et 
al., 1986a): each processing task is distributed across 
the network, and each single constituent neuron is 
involved in processing multiple tasks. Here, the statisti- 
cal contribution of each neuron’s variance to a given 
processing dimension (i.e. PC) is quantitatively defined 
as the square of the weighting for that PC. 


4.2. Functional attributes of PCs 


The functional attributes of each PC were reflected 
both in its weightings of neurons with different recep- 
tive fields, and also the physiological properties of the 
population vectors derived from such weights. The 
finding that most of the derived eigenvectors defined a 
different ‘topography’: ranging from the highly general- 
ized maps of the first few PCs, to the increasingly more 
selective mappings encoded by each subsequent PC. 
Although the sensory thalamus is commonly considered 
to contain a single receptor surface density map, PCA’s 
optimal mapping of the linear subspaces embedded in 
actual VPM neuronal activity was unable to resolve all 
such neural information into such a single two-dimen- 
sional topographic representation. In fact, the greatest 
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single source of variance (i.e. PC1) was non-topograph- 
ical, mainly encoding the magnitude of activity across 
the whole ensemble. In contrast, PC2 was typically 
topographical, but generalized it maximally across the 
neural population by encoding a linear rostrocaudal 
gradient. Subsequent PCs became ever more selective, 
often encoding sharp contrasts between adjacent 
whiskers, but also showing multiple peak/valley fea- 
tures within the same PC. The topographic specificity 
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of this organization of PCs is remarkable considering 
that they could be derived using data obtained either 
during sensory stimulation or spontaneous behavior 
which included exploratory whisking. In fact, these 
behavioral differences more often produced changes in 
the eigenvalues than the PC weighting patterns, there- 
fore changing their relative ordering. We conclude, 
therefore, that neural population activity in the VPM 
has an intrinsic and relatively invariant multidimen- 
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Fig. 9. PCs differentiate global from local information. In the VPM thalamus, spontaneous oscillations are ‘global’ in that most neurons are 
synchronized. In contrast, single whisker stimulation is local. (A) Spike rasters show activity of 23 simultaneous neurons over a 1.4-s period of 
an experiment involving intermittent E2 whisker stimulation, under awake conditions. (B) Eigenfunctions (PCs) | and 2 over this same time 
period. Eigenfunction | selectively indicates spontaneous global oscillatory peaks (e.g. at ‘O’), while eigenfunction 2 selectively indicates response 
(‘S’) to stimulation of whisker E2. (C) Peri-event histograms showing averaged responses of eigenfunctions (PCs) | and 2 to 306 repetitions of such 
stimuli during the same experiment. Stimuli consisted of 3° step deflections of the E2 whisker (100-ms duration, as shown by bar above). 
Responses to both the onset and offset of the stimuli are seen in both eigenfunctions. Both (B) and (C): Vertical axes depict average equivalent 
discharge rates of eigenfunctions: ticks, 10 Hz. Horizontal axes depict time (s). 
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sional factor structure which is maintained across dif- 
ferent waking behaviors. The behaviors themselves ap- 
pear to be largely encodable as modifications in the 
relative variances (i.e. activity levels) allocated to the 
different dimensions. 


4.3. Equivalent findings from computer simulations 


Interestingly, we have obtained very similar results 
by performing PCA on data obtained from a computer 
model of a simple sensory system (unpublished data). 
In this, a layer of neurons is fed by partially overlap- 
ping inputs from a linear matrix of receptors which are 
activated by moving stimuli. The PCs were observed to 
define a sequence of increasing spatial frequencies 
across the receptor surface, going from a single positive 
bell-type curve (PC1), to a gradient/half sinusoid (PC2), 
a full sinusoid (PC3), and so on. This sequence was 
found to reverse when the configuration of neuronal 
inputs was changed from partial-shifted overlap to cen- 
ter-surround inhibition: the weighting pattern originally 
exhibited by PC1 (with the highest eigenvalue) became 
that exhibited by the last PC (with the lowest eigen- 
value). Thus, the current findings argue that afferent 
responses in the VPM approximate a partial overlap- 
ping (‘distributed’) organization, rather than a center- 
surround (‘local’) organization. In the distributed-type 
organization, the highest eigenvalue components are 
constituted by small contributions of variance from 
large populations of neurons. Such components have 
the greatest opportunity to extract and generalize 
shared information across the ensemble. 


4.4. PCA, Hebb-type learning, and thalamocortical 
representations 


PCA’s detection of such generalized information 
emerging from the population of recorded neurons may 
have major implications for understanding how sensory 
input is processed, not only in the VPM, and also in its 
major target, the SI cortex. In fact, the known equiva- 
lence between PCA and general Hebbian learning sug- 
gests that it may be an especially powerful and 
functionally relevant method for characterizing the 
structure of sensory representations in brain regions 
which utilize Hebb-like adaptation for establishment or 
reorganization of synaptic connections. It is intriguing, 
therefore, to compare the weighting patterns of these 
PCs with our previously published findings about the 
diversity of shapes and sizes of receptive fields (RFs) in 
the whisker area of the rat SI cortex (Chapin, 1986). 
Using quantitative measurements, such RFs were ob- 
served to vary in size from single whiskers to the whole 
whisker field. Moreover, many of the RFs showed 
pronounced excitatory/inhibitory gradients between ad- 
jacent whiskers, or showed multiple peaks separated by 


inhibitory troughs. These cortical RFs are surprisingly 
similar to the topographies expressed by the different 
PCs defined here in VPM neuronal ensembles. As such, 
they could be produced by (polysynaptic) patterns of 
thalamocortical input weightings roughly similar to 
those derived here using PCA. The development and 
maintenance of such weighting patterns could result 
from Hebb-like (PCA-like) activity dependent adjust- 
ments of synaptic strengths in the SI cortex. 


4.5. Validity of PCA for understanding sensory 
representations 


The above observations lend some validity to the 
notion that the codes derived here using PCA may have 
some relation to ‘internal’ codes, as expressed either 
across neural populations or even within certain single 
neurons. It is reasonable to suggest that neuronal 
ensemble outputs could make weighted synaptic con- 
tacts on downstream neurons, and that those weights 
could be modified and maintained according to their 
covariance patterns, as in PCA and Hebb-type learning. 
Those neurons or populations could then provide a 
real-time ‘read out’ of a given PC, similar to the 
eigenfunctions constructed here. A major issue, how- 
ever, stems from the obvious fact that biological neu- 
rons have many nonlinear properties. To a certain 
extent, this is mitigated by our observation here that 
multi-spike train data from larger neural populations 
tend to converge on normal distributions from which 
linear approximations can be extracted. However, fur- 
ther progress may depend on improving our knowledge 
of the nonlinear aspects of the underlying information 
in these ensembles, and use of this knowledge to de- 
velop appropriate nonlinear PCA and PCA-like al- 
gorithms. Many such methods have been recently 
developed, especially in the field of artificial neural 
networks (Karhunen and Joutensalo, 1994). Unfortu- 
nately, the formal mathematical bases for nonlinear 
PCA is not yet sufficiently developed for it to be widely 
accepted in statistical analysis. For example, the major 
known advantage of nonlinear PCA appears to be its 
improved ability to separate independent signals from a 
noisy mixture (Karhunen and Joutensalo, 1994). Such a 
capability would have limited usefulness here: our aim 
is not to separate all of the individual signals impinging 
on a neuronal population, but to extract the informa- 
tional features produced by the association of these 
signals. 


4.6. Statistical resolution of PCA derived functions 


The statistical resolution of functionally important 
information by these PCs was very high, as determined 
by a number of measures. First, the S.E.s of the PC 
weights and eigenvalues tended to be very small frac- 
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tions of the total values. Second, the responses of the 
derived eigenfunctions to functionally significant events 
tended to be many S.D.s away from baseline activity. 
From the current results it is clear that the statistical 
resolution and ‘biasing’ of the information derived us- 
ing PCA is dependent on: (1) the total number of 
sampled neurons, (2) the time period over which they 
are recorded, (3) the size and complexity of the area 
sampled, and the homogeneity of sampling within that 
area, and (4) the sampling time integral, which col- 
lapses the temporal complexity of the spike train sig- 
nals. Theoretically, if more neurons were recorded over 
longer time periods using shorter time integrals, more 
significant local informational factors could be pulled 
out of the noise. 

One obvious distinction between the PCA approach 
used here and traditional PCA is that a relatively large 
number of components is needed to summarize most of 
the total variance. Here, only about 50% of the total 
variance was explained by the first 25% of the compo- 
nents. This fact, however, does not compromise PCA’s 
usefulness as a technique for reducing the dimensional- 
ity of complex information sets. Instead, it demon- 
strates the predictable result that neural activity in the 
VPM is relatively complex and high dimensional during 
waking behaviors. One could easily reduce this com- 
plexity by integrating over longer time bins or by 
averaging the data over multiple repetitions of the exact 
same stimulus event, but this is not desirable when the 
ultimate aim is to resolve the general functional reper- 
toire of this nucleus, with a fine spatial and temporal 
resolution. Moreover, since a relatively large propor- 
tion of the total variance of single neuron spiking is 
apparently ‘noise’ (i.e. is not shared with other recorded 
neurons) most of the total ensemble variance must 
inevitably be used to explain this noise. This fact under- 
scores the importance of using PCA to concentrate 
‘signal’ information in the lower numbered compo- 
nents, while consigning the noise to the higher num- 
bered components. 


4.7. Eigenfunctions 


The above results demonstrate that a range of func- 
tionally significant neural information can be resolved 
through eigenfunctions constructed by using PCA to 
provide weightings for averaging the activity of simulta- 
neously recorded neurons. In general, these provided a 
much greater resolution of neural events than could be 
obtained with data from any single neuron, and even 
greater resolution was obtained by combining multiple 
eigenfunctions. A major advantage of this increased 
resolution is that a number of significant sensory-be- 
havioral events could be reliably detected on a single 
trial basis using stripcharts of one or several eigenfunc- 
tions. This alleviates the necessity of averaging neu- 
ronal activity over multiple trials. 


An additional advantage is the ability of the PCA 
derived eigenfunctions to ‘focus’ on particular informa- 
tional features embedded in neuronal ensembles. This 
represents a major improvement over the use of simple 
unweighted ensemble averages (Dormont et al., 1982), 
which are only effective for encoding one dimension of 
information. Thus, combining the information from 
eigenfunctions 2 and 3 improved the ability to classify 
neural responses to stimulation of different whiskers, in 
fact mapping the whiskers roughly according to their 
topographic location on the face. The selectivity pro- 
vided by this multidimensional coding would not have 
been possible with the simple ensemble average. 

This ability of eigenfunctions to resolve significant 
sensory information in real time was especially useful 
for analysis of neural ensemble activity during sponta- 
neous behavior. In most cases, each of the first few 
eigenfunctions were robustly correlated with observable 
features of sensory-behavioral function, and these were 
predictable based on their PC weighting patterns. Thus, 
eigenfunction | tended to depict global, ensemble-wide 
functions, such as the overall magnitudes of sensory 
stimuli, regardless of their position or direction. In 
contrast, PCs 2 and 3 were more selective for whisker 
position, and more importantly, for specific directions 
of stimulus movement across the whisker field. Indeed, 
during whisking behavior these eigenfunctions tended 
to reveal the direction and speed of whisker movement 
across surfaces more than the identities of the specific 
whiskers which were touched. Finally, the higher num- 
bered components tended to encode information in 
higher spatial frequency domains, but more importantly 
defined specific relationships between neurons repre- 
senting particular nearby whiskers. To conclude, the 
eigenfunctions derived using PCA defined a functional 
topography between neurons. The degree to which this 
matched the anatomical topography between whiskers 
on the face was a function of the neural transformation 
of spatiotemporal patterns of whisker receptor activa- 
tion during spontaneous whisking. 

To a large extent, the lower numbered eigenfunctions 
express emergent, network-wide information, rather 
than information sequestered within small groups of 
neurons. Thus, these eigenfunctions allow measurement 
of information which exists mainly through small con- 
tributions of variance from each of a large number of 
neurons across the network. The resolution of such 
information is largely dependent on the number of 
neurons included in the population average. It can be 
predicted, therefore, that development of techniques for 
recording larger numbers of neurons in functional 
ensembles will allow increasingly fine resolution of the 
informational parameters involved in somatosensory 
processing. 

The fact that the eigenfunctions here, which were 
constructed using relatively small neural ensembles (8— 
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24 neurons) were able to detect, on a trial-to-trial basis, 
a number of significant behavioral events suggests that 
it might soon be possible to move beyond the classical 
neurophysiological requirement for averaging multiple 
repetitions of a stimulus. This capability is important not 
just for reasons of efficiency, but also because it is 
virtually impossible to create perfectly reproducible con- 
ditions for serial testing of neuronal properties. Not only 
are the behavioral conditions impossible to reproduce, 
but internal brain states which cause distinctive patterns 
of temporal coherence between neurons are even less 
controllable. Thus, network-based approaches such as 
that explored here could help alleviate the classical 
dependence of behavioral neurophysiology on over- 
trained animals. This would allow investigators to more 
precisely define the optimal behavioral tuning of neural 
activity by allowing it to be correlated with a much wider 
range of spontaneous and trained behaviors. In other 
words, if the experiment focuses on a narrow set of 
parameters chosen by the investigator, it cannot claim to 
have found the optimal property of the neurona. 

The ability to use PCA and/or other methods to 
encode multi-neuron population vectors marks an im- 
portant development in that it is now possible to resolve 
behaviorally significant brain information on a single 
trial basis. The next step is to extract such brain 
information in realtime and manifest it electronically in 
the real world. As an example, we have recently shown 
that PCA can be used to encode a realtime readout of 
32-neuron population activity recorded in the motor 
cortical forelimb area of rats trained in a lever position- 
ing task (Chapin et al., 1999). These rats were able to 
utilize such electronically manifested cortical ‘motor’ 
signals to directly control movement of a robot arm. 
Because of the accuracy imparted by using large neu- 
ronal populations, the animals were able to move the 
robot arm to fetch water from a dropper, and return it 
to their mouths using brain activity alone. 


4.8. PCA vs other strategies for depicting neuronal 
population information 


PCA is but one of a number of techniques which could 
be used to characterize neuronal population informa- 
tion. Since PCA is primarily descriptive, it does not 
depend on any preconceived theory about coding in the 
ensemble. It is significant, therefore, that the eigenfunc- 
tions derived here tended to be clearly and consistently 
correlated with externally observable sensory or motor 
phenomena. Nevertheless, there is no rule that each 
eigenfunction must necessarily code for a single specific 
identifiable behavioral event. For example, eigenfunc- 
tion 1 was found here to express overall magnitude of 
sensory stimulation, plus global 8—12 Hz oscillations 
during attentive immobility behaviors which were 
preparatory to onset of rhythmic whisker movement 
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(Nicolelis et al., 1995). Thus, the covariance mappings 
provided by PCA constitute opportunities to observe the 
information provided by functional groupings not antic- 
ipated by the experimenter. In fact, any close matching 
between eigenfunctions and target behaviors must be 
regarded as fortuitous. Correlated discharge between 
neurons can be caused by many internal and external 
factors, and need not necessarily match an experi- 
menter’s pre-existing hypothesis about coding in the 
system. For this reason it may be useful to combine PCA 
with techniques optimally designed for classifying 
groups, such as discriminant analysis, classification and 
regression trees (CART; Breiman et al., 1984), or neural 
networks with weights defined through supervised learn- 
ing techniques (Rumelhart et al., 1986b). This ‘hypothe- 
sis free’ aspect of the PCA is also different from 
population coding based on tuning functions (Georgo- 
poulos et al., 1986, 1988, 1989). 
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Abstract 


The goal of this study was to compare how multivariate statistical methods for dimension reduction account for correlations 
between simultaneously recorded neurons. Here, we describe applications of principal component analysis (PCA) and independent 
component analysis (ICA) (Cardoso J-F, Souloumiac A. IEE-Proc F 1993;140:362—70; Hyvarinen A, Oja E. Neural Comput 
1997;9:1483-92; Lee TW, Girolami M, Sejnowski TJ. Neural Comp 1999;11:417—41) to neuronal ensemble data. Simulated 
ensembles of neurons were used to compare how well the methods above could account for correlated neuronal firing. The 
simulations showed that ‘population vectors’ defined by PCA were broadly distributed over the neuronal ensembles; thus, PCA 
was unable to identify independent groupings of neurons that shared common sources of input. By contrast, the ICA methods 
were all able to identify groupings of neurons that emerged due to correlated firing. This result suggests that correlated neuronal 
firing is reflected in higher-order correlations between neurons and not simply in the neurons’ covariance. To assess the 
significance of these methods for real neuronal ensembles, we analyzed data from populations of neurons recorded in the motor 
cortex of rats trained to perform a reaction-time task. Scores for PCA and ICA were reconstructed on a bin-by-bin basis for single 
trials. These data were then used to train an artificial neural network to discriminate between single trials with either short or long 
reaction-times. Classifications based on scores from the ICA-based methods were significantly better than those based on PCA. 
For example, scores for components defined with an ICA-based method, extended ICA (Lee et al., 1999), classified more trials 
correctly (80.58 + 1.25%) than PCA (73.14 + 0.84%) for an ensemble of 26 neurons recorded in the motor cortex (ANOVA: 
P <0.005). This result suggests that behaviorally relevant information is represented in correlated neuronal firing and can be best 
detected when higher-order correlations between neurons are taken into account. © 1999 Elsevier Science B.V. All rights reserved. 


Keywords: Statistics; Independent component analyses; Neuronal ensemble interactions 


1. Introduction 


An emerging view in systems neuroscience is that 
behaviorally relevant information is represented by the 
concurrent activation of populations of neurons that 
form cell assemblies distributed across multiple levels of 
the nervous system (see Nicolelis et al., 1997b for 
review). The introduction of recording methods for the 
acquisition of spike trains from large populations of 
neurons (e.g. Kruger and Bach, 1981; McNaughton et 
al., 1983; Reitboeck, 1983; Nicolelis et al., 1997a; see 
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Nicolelis, 1999 for review) has intensified interest in the 
investigation of how distributed cell assemblies process 
behaviorally relevant information. A critical step for 
achieving this goal is the introduction of methods for 
data analysis that could identify functional neuronal 
interactions within the high dimensional data sets col- 
lected with these new recording methods. 

The initial basis of this study was the realization that 
multivariate statistical methods for dimension reduction 
might be able to reconstruct functional interactions 
between multiple neurons by providing a bin-by-bin 
estimate of correlated neuronal firing. Classically, di- 
mension reduction techniques have been used to trans- 
form a large set of observed variables into a smaller set 
of arbitrary variables, or factors, that are derived from 
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some property of the original data set. In the case of 
principal component analysis (PCA), the goal is to find 
a small set of linear functions that account for the 
covariance structure of the original high dimensional 
data set (see Chapin, 1999 for review). By contrast, 
recently developed methods for independent component 
analysis (ICA) attempt to isolate factors that are derived 
from higher-order correlations within the dataset, i.e. the 
higher moments describing the distribution of the vari- 
ables, such as kurtosis (see Comon, 1994 for review). 

Here, we report an application of ICA that allows for 
the visualization and quantification of behaviorally rele- 
vant information represented through correlated neu- 
ronal firing. To further validate this new approach, ICA 
was compared with a classical multivariate statistical 
method, principal component analysis, using data from 
(1) simulated ensembles of Poisson spike trains and (2) 
simultaneously recorded spike trains (n= 14-32) 
recorded in the rat motor cortex during the performance 
of a reaction-time task (Laubach et al., 1998). The main 
goal of this study was to investigate how PCA and ICA 
detect patterns of correlations across large ensembles of 
neurons. In addition, we show that these methods 
provide a quantitative representation of correlated neu- 
ronal firing that can be further analyzed with other 
methods, including techniques for statistical pattern 
recognition. 


2. Multivariate statistical methods 
2.1. Principal component analysis 


PCA is a classical method for multivariate data anal- 
ysis that describes interactions within a given data set 
that are derived from the matrix of correlations between 
all pairs of variables. PCA has been used for many years 
as a Standard method for preprocessing multivariate data 
sets to reduce their dimensionality. The goal of PCA is 
to find a reduced set of new variables that, in a decreasing 
order, account for the largest portions of covariance 
across the dataset. Further details about PCA can be 
found in textbooks on multivariate analysis (e.g. John- 
son and Wichern, 1992; Reyment and Jéreskog, 1993). 
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Fig. 1. In the blind source separation problem, one assumes that a set 
of measured signals, x, reflects the linear sum of some set of sources, 
s. The problem would be solved in theory if one could determine a set 
of weights, A, that are responsible for mixing the sources over the 
signals, i.e. the ‘mixing’ matrix. In practice, this is done by finding a 
different set of weights, B, that attempt to produce a new set of 
signals, y, that are as independent statistically as possible. 


Neurophysiological applications are described in 
Nicolelis and Chapin (1994) and Chapin and Nicolelis 
(1999). 

By design, individual principal components are unable 
to detect epochs of correlated neuronal firing between 
subsets of a neuronal ensemble. This is because the 
principal components are derived from the (second-or- 
der) correlation matrix for the neuronal ensemble. Thus, 
the first principal component is chosen so as to maximize 
the correlations between all neurons. Typically, this 
population vector is broadly mapped over the collection 
of neurons and is highly related to the average response 
of the neuronal ensemble (Nicolelis et al., 1998). Subse- 
quent population vectors are derived to account for 
maximal, orthogonal (co-)variance. However, the higher 
principal components (PC2 and higher), which often 
tend to account for the activity of subsets of neurons, are 
always dependent on the choice of the first component. 
Because the first component reflects the overall activity 
level of the neurons, the higher principal components are 
never able to clearly resolve independent sources of 
excitation from the overall activity of the ensemble. We 
illustrate this point below using data from simulated 
neuronal ensembles and from neurons recorded in the 
motor cortex of behaving rats. 

Varimax rotation (VMAX) is a modification of basic 
PCA that attempts to simplify the weight structure 
derived from PCA so that each component has a few 
variables with large coefficients and many with near-zero 
coefficients (Kaiser, 1958). The goal of VMAX is to 
adjust the coefficients for each principal component such 
that they are all near zero or near one. The results of this 
rotation is that the variance accounted for by each 
component is near maximal. A good account of the 
algorithm for VMAX is given in Reyment and Jéreskog 
(1993). 

In this paper, PCA was carried out using Matlab (The 
Mathworks, Natick, MA) and the Matlab statistics tool 
box. VMAX was done using code for Matlab as de- 
scribed in Reyment and Jéreskog (1993). 


2.2. Independent component analysis 


Several methods have been developed recently for 
ICA. These methods have been used mostly for the ‘blind 
source separation’ (BSS) problem (Fig. 1), where signals 
from a number of sources are mixed together in some 
visual or auditory dataset and the goal is to ‘un-mixed’ 
and recover the sources. The BSS problem is typically 
represented as: 


x(t) = As(t), (1) 


where x(t) represents the n observed signals (i.e. ne- 
urons) over ¢ time-points (i.e. bins), s(t) represents the 
n source signals, and A is the ‘mixing matrix’, that must 
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be resolved for source separation (see Comon, 1994 and 
Cardoso, 1998 for review). The main assumptions of 
the BSS problem is that the sources are independent 
and that they can be deduced using only the observed 
signals. This problem is reformulated to the following: 


y(t) = Bx(2), (2) 


where y(t) represents the statistical estimates of the 
sources that are derived from analyses such as ICA and 
B is the ‘unmixing matrix’ that is provided by a given 
algorithm for ICA. In general, ICA has been used 
under the assumption that the number of sources are 
equal to the number of signals. In dimension reduction 
applications, this assumption is no longer made. In- 
stead, one assumes the number of sources is less than 
the number of signals. 

Applications of ICA to the BSS problem work be- 
cause ICA resolves commonalties in the phase of the 
observed signals (Bell and Sejnowski, 1997), which are 
presumably due to the signals being influenced by com- 
mon sources. Phase relations between the observed 
signals are usually achieved by quantifying the kurtosis 
of the signals and finding factors that account for 
aspects of the signals that are ‘commonly peaked’, i.e. 
synchronous signals across multiple channels. In our 
application of ICA, PCA is used first for dimension 
reduction. Then, ICA is used to rotate the weights from 
PCA to make each component as independent as possi- 
ble. This is done by remapping the weights from PCA 
to maximize the kurtosis of each component. These 
signals invariably exhibit supra-Gaussian kurtosis, 
which implies that the signals are clustered around a 
range of values that is smaller than expected from a 
similar data set (i.e. with the same mean and standard 
deviation) generated from the Gaussian distribution. 
Theoretically, a remapping of PC weights that takes 
epochs of synchronous firing into account, should ‘un- 
mix’ independent signals that are represented by multi- 
ple PCs and produce a set of independent components. 

In the context of neuronal ensemble data, we inter- 
pret supra-Gaussian kurtosis in scores for the principal 
components as evidence for an independent brain sig- 
nal, such as a ‘common input’ in the sense of cross-cor- 
relation analysis. For example, if there are epochs of 
correlated firing that occur for a subset of the neuronal 
ensemble, these epochs will generate large scores in any 
principal component that maps onto the correlated 
neurons. When the PC weights are remapped to ac- 
count for the kurtosis of the PC scores, they will be 
adjusted such that a new set of weights is found that 
indicates which neurons tend to fire together. 

Two types of algorithms have been used to solve the 
ICA problem. Algebraic methods, such as the joint 
approximate diagonalization of eigenmatrices method, 
or JADE (Cardoso and Souloumiac, 1993), utilize ma- 
trix transformations to find a mapping over the vari- 


ables that maximize for their kurtosis. By contrast, 
neural network methods for ICA, such as those used in 
this paper (Hyvarinen and Oja, 1997; Lee et al., 1999), 
solve this problem by finding a set of weights that 
maximize the entropies of each independent component 
and minimize the mutual information between indepen- 
dent components, an idea that was proposed originally 
by Bell and Sejnowski (1995). 

The success of ICA for finding commonalities 
amongst multiple, simultaneously recorded neuronal 
spike trains, such as correlated firing, is directly related 
to ICA being based on analyses of higher-order correla- 
tions. For our application of ICA, we assume that the 
sources are brain-derived signals that alter the excitabil- 
ity of multiple neurons in our simultaneously recorded 
neuronal ensembles. In the simulations used in this 
paper, this relation was studied explicitly by adding 
spikes from the sources to the simulated spike trains. In 
data collected from the motor cortex, it is assumed that 
commonalties in the neurons’ response properties arise 
from common sources of inputs either from the cortex 
or subcortical areas (e.g. thalamus) that project to the 
motor cortex. Such sources of common input could 
produce nearly simultaneous increases in the firing 
probabilities of the neurons that receive their projec- 
tions. Thus, neurons that receive common inputs will 
have correlated responses, both around the time of the 
event that drives the source neurons and on a trial-by- 
trial basis. This point is illustrated below using simu- 
lated neuronal ensembles and neuronal ensembles 
recorded in the motor cortex of rats that performed a 
reaction-time task. 


3. Methods 
3.1. Simulated spike trains 


Simulated spike trains were created as follows: Back- 
ground discharge was created for 64 channels of data 
over 1000 ‘trials’. For each time-point in the spike train 
(an array with | ms bins for each time-point), a Poisson 
process was simulated with a rate of 1 Hz. If the 
generator produced a count, a spike was added to the 
bin. A minimum refractory period of 2 ms was used 
such that if a spike was placed in a given bin, then the 
Poisson generator skipped over the next two bins in the 
spike train. 

Four sources of excitation were generated as above 
except that the rate of the Poisson generator was set to 
between 50 and 200 Hz (with a single rate used for a 
given simulation). In addition, spikes were only gener- 
ated for a window of time, the ‘response window’, after 
the occurrence of a periodic ‘stimulus’ that occurred at 
100 ms intervals. The timing of the simulated neuronal 
responses was based on the response properties of real 
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Fig. 2. An example of the simulated neuronal ensembles. (a) The distribution of four sources over an ensemble of 64 neurons over 1000 trials. 
(b) Response properties of source 1 and four neurons. (c) Cross-correlational analysis for these neurons. 


cortical neurons studied in receptive field mapping ex- 
periments (A.A. Ghazanfar, personal communication). 
The response window began 8 ms after the stimulus and 
ended 15 ms later. The maximum probability of spiking 
within the window occurred at a latency of 12 ms. 

Spikes were added from the sources to nine neurons 
in the simulations of 64 neurons. For a given simula- 
tion, the correlation between the sources and the neu- 
rons was set to 0.2—0.8 (with a single correlation used 
for a given simulation). Source spikes were added for 
each neuron independently using a pseudo-random 
number generator based on the uniform distribution. 
Spike times from the sources were randomly jittered in 
some simulations by adding 0, 1, or 2 ms to the times 
of the sources spikes. If the neuron’s spike train con- 
tained a (background) spike at the time of the source 
spike, then the source spike was not added to the 
neuron’s spike train. Cross-correlation analysis was 
then used to show that neurons that shared a common 
source had peaks in their cross-correlation histograms 
that were more than was expected from the shift predic- 
tor (Perkel et al., 1967), which was determined using 
Stranger (Biographics, Winston-Salem, NC). By con- 
trast, neurons that received spikes from different 
sources had central peaks in their cross-correlation 
histograms, but these were not larger than expected 
from the shift-predictor test. 

These points are illustrated in Fig. 2, which shows 
data from one of the simulated neuronal ensembles. In 
panel (a), the distribution of the four sources’ influence 
over the ensemble of 64 neuron-like elements, which 


were arranged in a 8 x 8 matrix, is shown. In addition, 
the relation of four of the neurons to each source is 
depicted in the right part of panel (a). Raster plots and 
peri-event histograms for source 1 and four neurons 
from a typical simulation are shown in panel (b). The 
‘trigger event’ (time = 0) produced increased activity in 
source 1, which resulted in increased activity in neurons 
10 and 11. Neuron 55 had a similar response pattern 
but was driven by a different source (#4). Neuron 56 
was not driven by any of the four sources and so only 
contained random, background activity. Cross-correla- 
tion histograms for the neurons are shown in panel (c). 
The black lines represent the raw correlation between 
neuron 10 and itself (autocorrelation) and the other 
three neurons. The red lines represent the shift predic- 
tor, which accounts for correlations between neurons 
with similar response properties that are due to differ- 
ent sources of excitation. Note that the raw correlo- 
gram for the simulated neurons that were excited by a 
common source (#10 and #11) has a peak around 
zero lag that is larger than the peak in shift predictor. 
By contrast, the correlogram for neurons 10 and 55, 
which did not receive common inputs, is not larger than 
that of the shift predictor. 


3.2. Neurophysiological data 


Adult male rats were trained to perform a simple 
reaction-time task. In the task, the animals were re- 
quired to maintain a lever press over a variable interval 
(400—800 ms) and to release the lever within 1 s of the 
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onset of either a tone, vibration of the lever, or the 
combination of these stimuli. Correct performance of 
the task was rewarded by providing the rats with a 
drop of water 100 ms after each correct response. 
Incorrect responses occurred when the rats released the 
lever before the stimuli or had reaction-times longer 
than 1 s. These inappropriate behaviors were signaled 
by turning off the houselights for 2 s (i.e. time-out). 
After the animals demonstrated asymptotic perfor- 
mance in the task, arrays of microwire electrodes were 
implanted in the motor and premotor cortices using 
standard stereotaxic and aseptic procedures (Nicolelis 
et al., 1997a). 

The behavioral data were analyzed as follows: Reac- 
tion-times (RTs) were computed as the time from the 
onset of the trigger stimuli to the initial movement of 
the response lever, which was detected by a mi- 
croswitch. The upper and lower quartiles for the distri- 
butions of RTs were determined and used to partition 
the trials into those with short RTs (i.e. lower quartile), 
long RTs (i.e. upper quartile), and intermediate RTs 
(i.e. middle 50% of data). Quantitative analyses of the 
relationship between neuronal responses and RT be- 
havior were made by comparing trials with short and 
long RTs using the methods for PCA and ICA de- 
scribed in this paper and artificial neural networks that 
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were trained to discriminate between trials with RTs in 
the lower and upper quartiles. 


3.3. PCA and ICA 


The steps in our data analysis procedure are depicted 
in Fig. 3. The organization of the data was as follows: 
First, a matrix was constructed from the spike trains of 
the neurons (Fig. 3a). Each row in the matrix repre- 
sented neural activity from a single behavioral trial. 
Each column represented a time point (i.e. a bin of 1 
ms) in a spike train of one of the neurons. Sets of 
columns containing spikes for the individual neurons 
were appended together, to create the raw data matrix. 
This matrix had as many columns as the product of the 
number of neurons and the number of bins. The spike 
trains were smoothed and decimated five times, using 
low-pass filtering with a 3-tap Hamming window and 
decimating the spike trains with a 32-point Kaiser 
window. This procedure gave rise to an effective bin 
size of 5 ms. Filtering and decimation was done using 
routines in the Signal Processing toolbox for Matlab. 

The matrix of smoothed spike counts described 
above was then rearranged such that each neuron was 
represented as a column and each bin was represented 
as a row. Spike trains from individual trials were thus 
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Fig. 3. The data structures and analysis procedures used in this study. (a) Data structures for applying independent component analysis to 
neuronal ensemble data. (b) The series of steps in data analysis used in this study. 
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Fig. 4. Spike shifting was used to compare how well PCA and 
different methods for ICA accounted for correlated neuronal firing. 
Here, spike trains from a group of seven neurons are shown for the 
original condition (left) and after shifting (right). Each row in the 
plots represents spikes from a different neuron. Spike trains from 
each neuron and each trial were shifted by a random amount of time 
and the amount each neuron was shifted was chosen independently. 
The amount and direction of time shift for each neuron is indicated 
by the size of the arrow head to the left of the raster plots. 


appended together sequentially. The mean value for 
each column (i.e. for each neuron) in this matrix was 
subtracted and each column was then normalized to 
have unit variance. Ensemble averages were computed 
as the average response over the collection of rows 
representing each column (i.e. time-point or bin). Prin- 
cipal components for the zero-mean, unit variance ma- 
trix of smoothed spike counts were extracted using the 
singular value decomposition algorithm. The number of 
major components, that indicated the number of inter- 
acting groups in the ensembles, was identified by find- 
ing the number of eigenvalues that were larger than 1 
(for data with variance of 1). This value was also used 
to limit the number of independent components ex- 
tracted from the matrix of spike counts and to deter- 
mine the number of principal components that were 
rotated using VMAX. 

Logistic (IICA) and extended (xICA) ICA were run 
with principal component dimension reduction and 
data whitening (Lee et al., 1999). Fast ICA (Hyvarinen 
and Oja, 1997) was done with principal component 
dimension reduction and with sequential and simulta- 
neous extraction of the independent components using 
the following contrast functions: kurtosis (fICAk), tan- 
sig (fICAt), and the Gaussian non-linear function 
(fICAg). Scores for the principal and independent com- 
ponents were reconstructed on a bin-by-bin basis and 
the resulting matrices representing the components were 
re-arranged into a peri-event form (i.e. each row is a 
trial, and each column is a bin, with a block of x bins 
representing a component’s score with x bins per trial). 


Local features from the average peri-event his- 
tograms for the single neurons, the ensemble average, 
and the principal and independent components were 
identified using the wavelet-based method, discriminant 
pursuit (Buckheit and Donoho, 1995). These features 
were then further analyzed with an artificial neural 
network for statistical pattern recognition. The wavelet 
analysis was done as follows: The average response for 
trials with long reaction-times was subtracted from the 
average for trials with short reaction-times, to give a set 
of ‘difference vectors’ for each representation of the 
neuronal population. The difference vectors were then 
decomposed into a wavelet packet table using the 
Daubechies 4-point wavelet. Entries in the wavelet 
packet table that accounted for the largest amplitudes 
in the difference vectors were selected as local features. 
Scores for these features were extracted on a trial-by- 
trial basis and used as input to competitive neural 
networks, trained with a learning vector quantization 
algorithm (Kohonen, 1997; see Nicolelis et al., 1999 for 
further details on this method), that classified single 
trials as having either short or long reaction-times. 
Leave-one-out, or N-fold, cross-validation (Mosteller 
and Tukey, 1977) was used to assess error rates from 
this classification procedure. 

To test if precise timings between spikes across the 
group of neurons was an essential feature for PCA and 
ICA, a spike shifting procedure was used (Nicolelis et 
al., 1999). Spike shifting (see Fig. 4) was performed as 
follows: Spikes from each neuron were pseudo-ran- 
domly shifted in time between | and 7 bins, which gave 
perturbations in spike timing of between 10 and 70 ms. 
This was done to reduce the zero-lag coherence between 
neuronal spike trains on a trial-by-trial basis. These 
data were used as testing data for the artificial neural 
network that was trained with scores for the indepen- 
dent components obtained from the original, unshifted 
spike trains. The effect of the shift test was to perturb 
both the temporal precision of the neuronal firing pat- 
terns and the zero-lag coherence of the patterns. 


4. Results 
4.1. Simulated spike trains 


Analyses of the simulated neuronal ensembles 
showed that the ICA-based methods and varimax rota- 
tion of the principal components were able to segregate 
the sources to the appropriate neurons (Fig. 5). By 
contrast, standard PCA was unable to resolve the influ- 
ence of the individual sources on the ensembles. That is, 
VMAX and all methods for ICA weighted highly on a 
few neurons that were driven by a common source. By 
contrast, weights for PCA were broadly distributed 
over many neurons, with PC1 having weights for neu- 
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rons that were not even driven by the sources. This 
result indicates that the ICA methods and VMAX are 
better suited for detecting sparsely distributed func- 
tional relations within a neuronal ensemble. In other 
words, if information from one cortical area was con- 
veyed to a restricted portion of some neuronal ensem- 
ble, then the ICA-based methods would be able to 
detect the input and assign it to the appropriate neu- 
rons. This same input would be represented broadly by 
multiple PCs, thereby masking the interactions between 
neurons that were driven by the input of interest. 

The signals described from these weights are shown 
for some of the representations and are compared with 
the actual source signals in Fig. 6. Whereas scores for 
the PCs were effected by spikes in any of the four 
sources, scores for both JADE and xICA were more 
exclusively driven by the activity of a single source. 
This result was quantified by finding the Pearson corre- 
lation coefficients between the sources and the scores 
for functions derived with all methods. The maximum 
correlations between a single source and the compo- 
nents are shown on average in Fig. 7(a). Note that all 
methods except PCA were able to resolve at least one 
source (i.e. correlation coefficient large than 0.8). In 
addition, the ratios of the correlation between the com- 
ponents for each method and its two best sources are 
shown in Fig. 7(b). These values revealed that the 
ICA-based methods were more selective than either 
PCA or VMAX and that II1CA was somewhat worse 


1440 


than the other methods for ICA. These results imply 
that ICA-based methods identified the individual 
sources that drove the activity of subsets of the neu- 
ronal ensembles. By contrast, PCA represented correla- 
tions between neurons in a very different manner, 
which was too broad and did not reflect the nature of 
the underlying neuronal interaction. 

The shift test showed that the temporal precision of 
the correlated neuronal firing was no more than 20 ms 
(Fig. 8). That is, the ICA-based methods were selective 
to a given source only when the spike trains were 
shifted by less than 20 ms and these relationships were 
lost with further destruction of the temporal precision 
of the correlated neuronal firing. By contrast, PCA was 
not able to account for the source very well on any time 
scale and actually showed spuriously high selectivity for 
the source 1 when the spike trains were shifted more 
than 30 ms (see Fig. 8(c)). Thus, it appears that rela- 
tions between sources of excitation and neuronal re- 
sponses may only be valid on fairly precise time-scales, 
on the order of 30 ms or less. 


4.2. Neurophysiological data 


Ensembles of 14—32 neurons that were recorded dur- 
ing behavioral experiments in which rats performed a 
reaction-time task were used to extend the results above 
to actual neuronal ensemble data (Laubach et al., 
1998). The analyses were based on partitioning the 
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Fig. 5. Subsets of nine neurons in the simulated ensembles of 64 neurons received inputs from one of four sources (upper left plot). These 
distributions of source inputs were differentially identified and segregated by the PCA and ICA based methods. Note that the first principal 
component (PC) was broadly mapped over the entire neuronal ensemble, even on neurons that had only spontaneous background activity. The 
higher PCs appeared to segregate the different sources by contrasting the activity of different pairs of neuronal subsets. By contrast, the ICA-based 
methods, and also varimax rotation of the PCs, identified functional interrelations between the neurons and separated different sources to 
individual components. The plots in this figure are Hinton diagrams that depict the sign and magnitude of the coefficients using boxes of different 
sizes (i.e. magnitude) and color (i.e. sign). Black boxes represent positive coefficients and white boxes represents negative coefficients. 
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Fig. 6. ICA-based methods were better able to segregate the individual sources to individual components. The sources (upper left) and scores for 
three of the methods evaluated in the paper are shown over a segment of 500 ms. Note that scores for JADE (lower left) and xICA (lower right) 
were much more related to the single sources than were the scores for any of functions defined by PCA (upper right). 


collection of reaction-times for all trials into quartiles 
and comparing trials in the lower quartile with trials in 
the upper quartile. The goal of the study was to investi- 
gate if the rats’ reaction-times on single trials could be 
predicted by the firing of neurons in the caudal and 
rostral forelimb areas of the motor cortex. Neuronal 
activity over the 250 ms before and 100 ms after the rat 
released a response lever (a behavior that defined the 
operant response in the task) was analyzed with the 
various methods for PCA and ICA described above. 

The different methods produced very different ac- 
counts of neuronal interactions across simultaneously 
recorded neuronal ensembles from the rat motor cortex. 
The data set shown in Fig. 9 contained 26 neurons, 
with 18 neurons in the caudal forelimb area and six 
neurons in the rostral forelimb area. There were seven 
eigenvalues larger than | for this ensemble of neurons. 
In Fig. 9, coefficients are shown for PCA, varimax 
rotation of the coefficients from PCA (VMAX), and for 
two ICA-based methods, JADE and xICA. Whereas 
the coefficients defined through PCA and VMAX were 
broadly mapped over the neuronal ensemble, coeffi- 
cients from JADE and xICA were much more sparsely 
distributed over the neuronal ensembles. The popula- 
tion vectors defined by these different methods were 
based on different levels of correlated neuronal firing. 
PCA-based methods accounted for broadly distributed 
neuronal correlations and ICA-based methods ac- 
counted for more restricted interactions between the 
neurons. 

The ICA-based methods were able to more clearly 
segregate the response properties of the neurons during 


the time around the lever release for trials with short 
reaction-times (solid, dark lines) and those for trials 
with long reaction-times (dashed, light lines). As is 
shown in Fig. 10, scores for PCA and VMAX were 
highly similar despite the differences in the coefficients 
defined by these two methods (Fig. 9). By contrast, the 
average scores for population vectors identified with the 
ICA-based methods were somewhat different. For ex- 
ample, while the second component from xICA was 
highly similar to the third principal component, only 
the ICA-based method shows a peak on trials with 
short reaction-times while PCA shows peaks for both 
types of trials. When scores for population vectors 
defined with PCA, VMAX, and the ICA methods were 
used to train ANNs to predict the animals’ reaction- 
times on single trials, the methods gave differential 
predictions of trial outcomes. The best results were 
achieved using scores for independent components 
identified with xICA (80.58 + 1.25%), which classified 
more trials correctly than scores for PCA (73.14+ 
0.84%) (Fig. 11; ANOVA: P < 0.005). This result sug- 
gests that the higher-order correlations between these 
neurons, which were best resolved by xICA, conveyed 
behaviorally relevant information. 

Finally, when the shift test was used for these data, 
we observed that the variances of the scores for the 
population vectors were reduced (Fig. 12(a)). More- 
over, the degree to which mean signals for trials with 
short and long reaction-times differed was reduced and 
certain aspects of the response properties were lost 
altogether (Fig. 12(b)). The time-scale of this neuronal 
shift was over +3 bins, or 30 ms. Therefore, these data 
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show that behaviorally relevant information could be 
represented in correlated neuronal firing in the rat 
motor cortex and that this information is based on a 
synchronization of the spike trains on a relatively short 
time-scale and not on the overall excitability of the 
neurons. These findings indicate that theoretical models 
of reaction-time performance should consider correla- 
tions between subsets of the neuronal populations in 
motor cortex, in addition to traditional firing rate cod- 
ing, as a means of representing motor performance. In 
this context, our data support with recent studies that 
have implicated correlated neuronal activity in the sen- 
sorimotor cortex of behaving mammals in representing 
behaviorally relevant information (Murthy and Fetz, 
1992; Nicolelis et al., 1995; Seidemann et al., 1996; 
Riehle et al., 1997; Donoghue et al., 1998; Hatsopoulos 
et al., 1998). 
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Fig. 7. The ability of the different PCA- and ICA-based methods to 
account selectively for a single source of common input are shown. 
(a) The average maximum correlations between each source and each 
component are shown for eight methods. All of the ICA-based 
methods and varimax rotation of the principal components were 
related to single sources with correlation coefficients around 0.8. By 
contrast, the maximum correlations for the principal components 
were much lower. (b) The ratio between the maximum correlation 
between each source and each component and the next largest 
correlation are shown. This is a measure of the selectivity of the 
method to resolving a single source of common input. All ICA-based 
methods were more selective than both PCA and varimax rotation of 
PCA. 
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Fig. 8. The shift test produced degradations in the correlations 
between sources, neurons, and population vectors that were depen- 
dent on the range of time over which spikes were shifted. (a) 
Correlations between one of the simulated neurons, # 10, from one 
simulated data set and source | are shown for spike shifting over 10, 
30, 50, and 70 ms. The correlation between the neuron and the source 
rapidly degraded as the time-scale of shifting was increased. (b,c) The 
average maximum correlation between population vectors and source 
1 and for the ratio of the maximum correlation to the median 
correlation are shown for these same time-scales of shifting and for 
PCA and three ICA-based methods (JADE, xICA, and fICAg). The 
inset panel in (c) shows the ratio data on an expanded scale to better 
illustrate the differences between the three ICA-based methods. 


5. Summary and discussion 


Neural assemblies are thought to arise when neurons 
are dynamically arrayed into groups to convey and 
process information in a cooperative manner. This form 
of neuronal processing is based on the correlated time 
structure of individual spike trains. Though the time 
scale in which neurons can form assemblies is still a 
matter of debate (Abeles, 1991; Shadlen and Newsome, 
1994; Softky, 1995; Konig and Engel, 1996; Shadlen 
and Newsome, 1998; Nicolelis et al., 1999), analyses of 
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Fig. 9. Different methods for multivariate dimension reduction produced very different spatial distributions of neuronal interactions across 
simultaneously recorded neuronal ensembles from the rat motor cortex. This data set contained 26 neurons, with 18 neurons in the caudal forelimb 
area and six neurons in the rostral forelimb area. There were seven eigenvalues larger than | for this ensemble of neurons. Coefficients for these 
seven functions are shown for PCA, varimax rotation of the coefficients from PCA (VMAX), and for two ICA-based methods, JADE and xICA. 
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Fig. 10. These plots are scores for population vectors defined with PCA, varimax rotation of the coefficients from PCA (VMAX), and two 
ICA-based methods, JADE and xICA. The average scores for trials with short reaction-times are shown as solid, dark lines and those for trials 
with long reaction-times are shown as dashed, light lines. The reference event (time = 0) is the moment when the rats released the response lever 
within 500 ms of the onset of the trigger stimuli. 
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Fig. 11. More information about reaction-time performance was 
available from population vectors defined by the ICA-based methods 
than was by PCA or varimax rotations of the principal components. 
This figure shows the results of using scores for several of the 
dimension reduction methods to train artificial neural networks to 
discriminate between trials with short and long reaction-times based 
on the activity of 26 neurons in the rat motor cortex. Scores derived 
with xICA discriminated significantly more trials correctly than did 
scores derived with any other method (ANOVA: P <0.01). 


simultaneously recorded neurons all center on defining 
the correlational structure between spike trains under 
the null hypothesis that N neurons fire independently. 
After correcting for changes in individual spike rate 
modulations, pair-wise and population analyses seek to 
quantify any residual correlations that may indicate the 
presence of cell assemblies. A general limitation to such 
correlational analyses is that, while some sort of time 
averaging is required to estimate spiking probabilities, 
spike rates of individual neurons are known to be 
non-stationary. An increasing number of analyses ap- 
plicable to neuronal ensemble recordings have been 
developed that address this limitation, leading to an 
array of methods, each of which have various strengths 
and weaknesses. 

Cross-correlational analyses seek to establish a func- 
tional relationship between the firing times of one neu- 
ron with respect to that of another neuron; the 
principle method of which has been the construction of 
the cross-correlation histograms (CCH) (Perkel et al., 
1967). In the classical CCH, spike timing differences 
between the ‘reference’ spike train and the ‘target’ spike 
train are constructed without respect to the stimulus 
cycle. To normalize for stimulus induced changes in 
firing rates between the two neurons, the ‘shift predic- 
tor’ is subtracted from the CCH to remove any stimu- 
lus-locked activity. Alternatively, one can use the joint 
peristimulus time histogram (Aersten and Gerstein, 
1989) to determine the temporal structure of correlated 


firing between two neurons with respect to an external 
event. More recently, unitary event analysis has been 
proposed to detect correlations between pairs of neu- 
rons that need not be time locked to the occurrence of 
an external event (Riehle et al., 1997 and also in this 
issue). 

While pair-wise analyses are sensitive to determining 
the functional relationship between two neurons and 
some event, the major limitation to such analyses arises 
from the combinatorial explosion of paired-combina- 
tions needed to understand the inter-relationships 
between even a small ensemble of simultaneously 


Oo. + pil 


: J 
Ctrl Shift Ctrl Shift 


Normalized variance 


Control (Ctrl) er enien® shifting (ISh) 


b 1 

art 

$2 

- 

cal 

% ob 

i ee acre -1 1 1 1 1 1 J J 
% -200 -100 0 100 -200 -100 0 100 


Time (msec) 


— =ShortRT ------ = Long RT 


Fig. 12. Spike shifting reduced the variance of the scores for the 
population vectors and eliminated important aspects of the responses 
of the population vectors that discriminate between trials with differ- 
ent reaction-times. (a) The variance of the scores for the population 
vectors defined by ICA-based methods was reduced more than that of 
population vectors defined by PCA. This was due to the loss of 
correlated neuronal firing after shifting the spike trains. The data 
shown in the figure are from spike shifting over 10 ms epochs. (b) 
Averaged responses for population vectors defined by ICA-based 
algorithms discriminated between trials with different reaction-times. 
This discriminative information was lost after spike shifting, which 
produced decreases in the degree to which the average signals for 
trials with short and long reaction-times differed. For example, the 
second xICA shown in this figure exhibited an epoch of correlated 
firing approximately 50 ms prior to lever release on trials with short 
reaction-times and not on trials with long reaction-times. This peak 
was not apparent in the averaged histograms after spike shifting. 
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recorded neurons. As methods for recording from 
ensembles of neurons improve and the number of 
simultaneously recorded neurons continue to increase, 
the need for more appropriate methods for data analy- 
sis becomes even more apparent. 

Other methods that go beyond pairwise interactions 
have been developed: gravitational clustering (Gerstein 
and Aertsen, 1985), spike pattern search algorithms 
(Abeles and Gerstein, 1988), and hidden Markov mod- 
els (HMM) (Gat and Tishby, 1993; Radons et al., 1994; 
Seidemann et al., 1996). Gravity analysis is a popula- 
tion technique that treats all the neurons of the 
recorded ensemble as a single entity, clustering neurons 
based on synchronous firing. The typical ‘read-out’ for 
this method, however, is based on correlations between 
pairs of neurons. Spike pattern search algorithms allow 
for the detection of precise, reoccurring spike patterns 
across neurons and time to be detected and identified. 
But, by design, these methods do not provide trial-by- 
trial estimates of the detected patterns and so they do 
not allow for quantification of information represented 
in the identified patterns. On slightly broader time 
scales (i.e. tens of milliseconds), HMMs have been 
applied to detect concomitant firing-rate modulations 
between members of the recorded neuronal ensemble. 
The goal of these analyses is to determine the variety of 
firing patterns exhibited across an ensemble of neurons 
(Seidemann et al., 1996). Thus, HMMs appear to an- 
swer very different questions about correlated neuronal 
firing than those addressed by the other methods de- 
scribed above and in this paper. 

Another approach to the problem of correlated neu- 
ronal firing was taken by Nicolelis and Chapin (1994) 
when they applied multivariate statistical methods for 
dimension reduction, such as PCA, to neuronal ensem- 
ble spike train data. The major advantage of this 
approach over the cross-correlational and pattern anal- 
ysis methods described above is that the method explic- 
itly provides a representation of correlated neuronal 
firing on a bin-by-bin and trial-by-trial basis. This 
allows for quantitative estimates of the degree to which 
correlated neuronal firing, as represented by PCA or 
ICA, may vary with the behavioral performance of an 
animal. Moreover, it has been shown recently that these 
signals can be used to control external devices such as 
robotic arms (Chapin et al., 1999). Thus, in addition to 
having theoretical applications in determining the types 
of neuronal codes that might arise through neuronal 
ensemble interactions (e.g. Laubach and Nicolelis, 
1998), multivariate statistical methods have great po- 
tential for practical applications in _ clinical 
neurophysiology. 

Here, we compared how PCA- and ICA-based meth- 
ods detected and reconstructed epochs of correlated 
firing across ensembles of simultaneously recorded cor- 
tical neurons. We observed that while PCA was able to 
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resolve a variance-weighted version of the average sig- 
nal across the ensemble, it was unable to identify 
sparsely distributed interactions across the ensembles. 
By contrast, each of the methods for ICA that was 
examined was able to account for sparsely distributed 
interactions. Therefore, it appears that PCA is an excel- 
lent method for accounting for the overall excitability 
and broad correlations between large groups of neu- 
rons. By contrast, the ICA-based methods used here are 
better suited for finding higher-order correlations (i.e. 
synchronous firing) between the activity of more limited 
portions of a neuronal ensemble. 

Several methods for ICA were compared in this 
study. For the most part, the different algorithms pro- 
duced similar results, despite the differences in their 
underlying nature (i.e. algebraic versus iterative) or the 
form of the contrast function used (e.g. kurtosis, tansig, 
Gaussian). The simulations showed that the selectivity 
of the various methods was roughly equivalent, with 
the major differences being in the range of variability 
exhibited by different methods. By contrast, our appli- 
cation of these methods for real neuronal ensemble 
data found that the xICA algorithm performed better 
than the other methods. This may be due to the nature 
in which the xICA network is updated, which is differ- 
ent from the other methods. Prior to xICA, the data 
series are randomly permuted such that the order of 
bins is not longer that obtained in the original data set. 
Thereafter, the network is ‘trained’ on randomly chosen 
time epochs that are arranged in frames with as many 
elements as there were bins on a single trial. Thus, the 
xICA algorithm iterates through various instances of 
the neuronal data in a way that should overcome any 
explicit episodes of non-stationarity in the data set. 
This is not the case for the JADE and fast ICA 
algorithms, which operate on the data in ‘batch’ mode, 
with all bins used to estimate the weights for the ICA 
model. It may be that the xICA procedure overcomes 
non-stationarities in the neuronal data that limited the 
success of the other methods. Given that the limited 
amount of data in the example used in this paper (70 
trials) is actually typical of most real behavioral neuro- 
physiological data sets, we suspect that xICA maybe 
the best choice for future applications of ICA-based 
methods for analyzing neuronal ensemble data. 

Despite the advantages of ICA over cross-correla- 
tional methods, the method has a number of limita- 
tions. Our experience with ICA has shown that it will 
work appropriately when the following conditions are 
met: (1) firing probabilities are relatively low; (2) corre- 
lations within sub-ensembles are relatively high; (3) the 
temporal precision of the neuronal spike trains is high. 
For example, if several independent sources for neu- 
ronal synchronization effect activity in a given subset of 
a neuronal population (1.e. spatial overlap of multiple 
sources), then a given independent component may 
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represent a linear combination of the sources rather 
than a true independent source (Makeig et al., 1999). 
Such a situation is likely to arise in most network 
level studies of brain function, suggesting that more 
work is needed to further improve methodologies for 
analyzing the high-dimensional data sets that are rou- 
tinely acquired using modern methods for neuronal 
ensemble recording. 

The techniques described in this paper may soon be 
extended to address a genuinely new question for ex- 
perimental neuroscience: the role of interactions be- 
tween multiple brain areas. For example, if arrays of 
recording electrodes are placed in several cortical ar- 
eas that are know to be connected anatomically, then 
the methods for ICA used in this paper could be 
applied to each area to find a set of components that 
account for correlated firing within each cortical area. 
Statistical pattern recognition methods could then be 
used to evaluate the contributions of the individual 
cortical areas to the collective representation of the 
animal’s behavior, such as in a reaction-time task. In 
addition, other methods, such as the method of di- 
rected coherence (Sameshima and Baccala, 1999) 
could be used to determine the direction of informa- 
tion flow between multiple cortical networks using a 
reduced set of variables that are provided by the in- 
dependent components. The exploration of these is- 
sues, especially when applied to simultaneous 
recordings from multiple areas of cortex, is likely to 
become one of the main topics of research in systems 
neuroscience. 
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A paradigm is described for recording the activity of single cortical 
neurons from awake, behaving macaque monkeys. Its unique 
features include high-density microwire arrays and multichannel 
instrumentation. Three adult rhesus monkeys received microwire 
array implants, totaling 96-704 microwires per subject, in up to five 
cortical areas, sometimes bilaterally. Recordings 3-4 weeks after 
implantation yielded 421 single neurons with a mean peak-to-peak 
voltage of 115 + 3 wV and a signal-to-noise ratio of better than 5:1. 
As many as 247 cortical neurons were recorded in one session, and 
at least 58 neurons were isolated from one subject 18 months after 
implantation. This method should benefit neurophysiological in- 
vestigation of learning, perception, and sensorimotor integration 
in primates and the development of neuroprosthetic devices. 


he use of single, movable electrodes to record neuronal 

activity in macaque monkeys has proven to be an extremely 
productive method in neurophysiology. Although the method 
was originally developed by Evarts (1) to study sleep, he later 
noticed that when monkeys moved, neurons in motor cortex 
discharged “with... frequencies ... reaching 60-80 per sec at 
certain phases of movement.” Thus, neurophysiologists could 
study activity in awake, behaving monkeys with methods that 
have changed little in the past 40 years. 

Although Evarts’ method was a breakthrough for neurophys- 
iology, the use of single, movable electrodes typically limits data 
collection to one or two neurons at a time for brief periods. To 
study a neuronal population, cells must be sampled serially, over 
weeks. Newer systems use several movable electrodes, typically 
6 to 16, but recordings remain limited to one cortical area with 
relatively brief recording durations. These factors impede the 
study of neural correlates of learning, neuronal interactions, and 
population activity. 

Long-term, multielectrode recordings have become routine in 
rodent neurophysiology (2-6) and have recently been adapted to 
New World monkeys (7, 8). However, robust and reliable 
application of chronic, multiple-electrode recording methods in 
macaque monkeys remains a challenge. Meeting that challenge 
is important because macaques have been the primate model of 
choice for studying the neural mechanisms underlying percep- 
tion, movement, control, and cognition. 

Recent attempts to obtain long-lasting, single-neuron record- 
ings from macaques have used the 100-electrode “Utah array” 
or arrays of individual microwires (9-14). However, these studies 
have provided modest neuronal yields of uncertain longevity, 
and, in most cases, have been limited to just one or two cortical 
areas per animal. This article describes a paradigm for recording 
simultaneously from a larger population of neurons in a larger 
number of cortical areas for up to 18 months. It also describes 
methods for evaluating the quality of single-unit isolation in 
multielectrode recordings. 


Methods 


All recordings described in this study were obtained through 
chronically implanted arrays of microwires. In one animal, we 
used commercially available (NBlabs, Dennison, TX) low- 
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density arrays formed by 16-32 50-um Teflon-coated stainless 
microwires (15). Two other animals were implanted with high- 
density microwire arrays developed at Duke University. The 
main design used in this study consists of: (7) an array of S-isonel 
(or Teflon)-coated tungsten (or stainless steel) microwire elec- 
trodes, (ii) a printed circuit board (PCB) connected to the 
microwire electrodes, and (iii) a high-density, miniature con- 
nector attached to the opposite side of the PCB (see Fig. 1 A-C). 
The flexibility of our PCB design allows virtually any type of 
microwire to be used as an electrode. 


Subjects. Three rhesus monkeys (Macaca mulatta) were used in 
this study: two females (monkey 1, 9.7 kg and monkey 2, 9.9 kg) 
and one male (monkey 3, 10.2 kg). All procedures conformed to 
the National Academy Press Guide for the Care and Use of 
Laboratory Animals and were approved by the Duke University 
Animal Care and Use Committee. 


Surgery for Implantation of Microwire Arrays. Aseptic surgical 
technique was used throughout. Deep surgical anesthesia was 
maintained throughout the procedure by using a combination of 
isoflurane and iv. fentanyl. After several craniotomies were 
drilled, microwire array touched down the surface of the brain 
under visualization by using the operating microscope (Fig. LD). 
The electrode array was advanced slowly at ~100 um per 1-min 
intervals by using a hydraulic micropositioner (David Kopf 
Instruments, Tujunga, CA). Penetration of the brain was as- 
sessed electrophysiologically by isolation, amplification, and 
visual and audio monitoring of neuronal activity. Up to 5 to 10 
cortical implants were made per animal (Fig. 1 E and F). 


Multisite, Many-Neuron Ensemble Recordings. A multichannel ac- 
quisition processor (MAP, Plexon, Dallas) cluster, formed by 
three 128-channel MAPs synchronized by a common (2 MHz) 
clock signal and an external start pulse, was specially built for the 
experiments reported here. This 384-channel recording system 
has a theoretical capacity of recording up to 1,536 single neurons 
simultaneously (e.g., four neurons per channel), at 25-pus 
precision. 


Single-Unit Spike Sorting. The first step in all recording sessions 
involved the setting of a voltage threshold for each of the MAP 
channels processing signals from the implanted microwires. This 
threshold was set by the experimenter through visual inspection 
of both the original analog signal (displayed in an oscilloscope) 
and the digitized record displayed on the screen of the computer 
controlling the MAP. All events that crossed the set voltage 
threshold (defined as unsorted waveforms) for each channel 


Abbreviations: MAP, multichannel acquisition processor; PC, principal component; ISI, 
interspike interval; PsF, PseudoF; DB, Davies-Bouldin; Vpp, peak-to-peak voltage; $1, 
primary somatosensory cortex; M1, primary motor cortex; PMd, dorsal premotor cortex; 
SMA, supplementary motor area. 
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Fig. 1. (A) Printed circuit board (PCB) used to build a high-density microwire 
array (spacing between traces is 300 wm). (B) Multiple boards are stacked up 
to form arrays with up to 128 microwires. (C) High-density connectors are 
attached to the other side of the board. (D) Implantation of a 128-microwire 
array in the M1. (E) Last stages of a 672-microwire implant. Bilateral implants 
included the PMd, M1, and the PP. (F) Final aspect of a multisite implant after 
the application of multiple layers of dental cement. 


were digitized and recorded continuously for up to 2-3 h for 
subsequent off-line spike sorting analysis. An Off-Line Spike 
Sorter (OFSS, Plexon) was used to isolate single units from all 
our recording sessions. Initially, principal component (PC) 
analysis was carried out by using all unsorted waveforms re- 
corded. Results involving the first eight PCs were made available 
to the user, because they usually accounted for close to 90% of 
the variance of the waveforms. Next, the experimenter selected 
three of the available components to define 2D/3D PC spaces. 
These PC spaces were used to plot the values of the correspond- 
ing PC scores calculated for all unsorted waveforms. Inspection 
of clusters of points observed in PC space took advantage of 
several tools provided by OFSS. These tools allow the experi- 
menter to visualize the waveform associated with each point in 
the 2D/3D PC space, measure the waveform’s peak-to-peak 
voltage amplitude, and compare it with events immediately 
before or after its occurrence. This latter option involved the use 
of a “digital oscilloscope” that provided a continuous record of 
all of the stored events from a given channel. By selecting a group 
of points from the PC space, interspike interval (ISI) histograms 
were calculated. All of these tools were used to identify whether 
the channel contained any spurious signals (e.g., electrical noise, 
movement artifact). Spurious signals were classified as invalid 
spikes and removed from the PC space. The remainder of the 
waveforms, denominated valid spikes, were then scrutinized for 
the isolation of single units. 

Based on the quantification of multiple parameters (waveform 
voltage, waveform shape, ISIs, etc.), the experimenter then 
defined the cluster partition for each channel. This was done by 
manually defining the clusters shown to contain similar wave- 
forms (see an example of the off-line spike sorting procedure in 
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Fig. 2. Examples of cluster separation, 30 days postimplantation. Each 
example contains: (Left) identified clusters in 3D PC space; (Upper Middle) 
waveforms; (Upper Right) ISI histograms; (Lower Right) continuous record of 
potentials. Different colors depict distinct clusters in all parts of each example. 
J3 (2D), F (2D), and DB (2D) statistics are at the far right. (A) Three single units 
(yellow, green, and blue) discriminated from the multiunit record (red cluster) 
yielded high J3 and F values, average DB, clear differences in waveforms, and 
ISI histograms. (B) Two single units (green and yellow) were differentiated 
from the multiunit activity (blue), producing high J3 and F and low DB values. 
(C) High separation between the multiunit cluster (blue) and two units (yellow 
and green). (D) One large (yellow) and one small (green) unit were isolated 
from the multiunit activity (blue). (E) Two clearly distinct units were isolated. 
J3 was above the mean, DB was good, and F value was average. (F) Two units 
with distinct amplitudes and ISI histograms from a channel with below aver- 
age J3 and F values. The DB value indicates good cluster separation. (G) 
Channel with low J3 and F values, but an average DB. Two units with distinct 
waveforms and ISI histograms were isolated. (H) Example of a large single unit 
(yellow) isolated from the multiunit activity (green). J3 was below average, F 
was very high, and DB was very low. (/) Single unit (yellow) clearly distinct from 
the multiunit activity (green). The J3 value was close to its mean, the F value 
was high, and DB was low, indicating excellent cluster separation. (J) A single 
unit (yellow cluster) isolated way above the multiunit voltage level. ISI histo- 
gram time scales were 5 ms (C), 10 ms (A, D, and G), and 100 ms (B, E, F, and H-J). 


Fig. 6, which is published as supporting information on the PNAS 
web site, www.pnas.org). 

A group of similar waveforms was considered as being gen- 
erated from a single neuron only if it defined a discrete cluster 
in 2D/3D PC space that was distinct from clusters for other units 
and/or multiunit activity. In addition, single units had to exhibit 
a clearly recognizable refractory period (>1 ms) in their ISI 
histograms and have a characteristic and distinct waveform 
shape and peak-to-peak amplitude when compared with other 
neuronal waveforms and multiunit activity (as seen in the digital 
oscilloscope) in the same channel (see Fig. 2). 

Four different statistics were used to objectively quantify the 
overall separation between identified clusters in a given record- 
ing channel. These measurements included the classic paramet- 
ric F statistic of multivariate analysis of variance (MANOVA), 
the J3 and PseudoF (PsF) statistics (16, 17), and the Davies- 
Bouldin (DB) validity index (ref. 18 and see Supporting Text, 
which is published as supporting information on the PNAS web 
site, for details on how these statistics were calculated). Control 
values for the four statistics were obtained from channels in 
which no single-unit activity was detected. To calculate control 
values of the four statistics, the single spheroid cloud of points 
centered at the origin of the PC space was divided in half. Then, 
two separate clusters with similar shape and number of wave- 
forms were defined by the experimenter so that the same 
statistics (F, J3, PsF, and DB) could be calculated. 

Mean signal-to-noise ratios were also calculated for all single 
units isolated by dividing the mean peak-to-peak voltage (Vpp) 
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of the isolated single unit by the mean “noise” level observed in 
our microwire recordings. Mean noise was calculated in a large 
sample of channels by lowering the voltage threshold to zero and 
then measuring the Vpp of the lowest amplitude waveforms that 
could not be differentiated into single units (the so-called 
multiunit activity). Short-term single-unit isolation stability (i.e., 
stability over hours) was evaluated with Wavetracker (Plexon). 
Data collected during 1- to 2-h recording sessions were first 
divided in consecutive 30-min segments. For each of these data 
segments, single units were discriminated, and 2D/3D PC clus- 
ters, waveforms, ISIs, and cluster separation statistics were 
calculated for each recorded channel. 


Results 


Because all animals continue to be studied, the microwire-array 
locations reported here are based on stereotaxis, cortical land- 
marks observed during surgery, and high-resolution digital RX 
imagery (in two animals), rather than histological methods. 

The mean impedance of the microwires implanted in our mon- 
keys was 1.5 MOhms (measured in saline at 1 KHz). In monkey 1, 
96 microwires were implanted in five locations: 16 in area 1 of the 
right primary somatosensory cortex (S1), 16 in each primary motor 
cortex (M1), 32 in the right dorsal premotor cortex (PMd), and 16 
in the right supplementary motor area (SMA). In monkey 2, 672 
microwires were implanted in three different cortical areas in each 
hemisphere: M1 (96 in each hemisphere), PMd (96 in the left 
hemisphere, 128 in the right), and posterior parietal cortex (PP, 128 
in each hemisphere). In monkey 3, a total of 704 microwires were 
implanted in 10 cortical locations, distributed in both hemispheres: 
PMd (96 in the left hemisphere, 64 in the right), M1 (96 in the left, 
64 in the right), S1 (64 in the left and 96 in the right hemisphere), 
PP (32 in the left hemisphere), and extrastriate visual cortex, near 
the boundary between the parietal and occipital cortex (96 in each 
hemisphere). 

As in New World monkeys (7, 8), the number of units isolated 
seemed to stabilize 4-6 weeks postsurgery. High-quality single 
units could be isolated and recorded 30 days after the implan- 
tation surgery. Fig. 2 illustrates the entire range, in terms of 
quality of single-unit isolation, of single units obtained in our 
recordings. Each example in Fig. 2 contains: color-coded cluster 
in PC space (left half of each lettered part of the figure), samples 
of the corresponding waveforms and ISI histograms from clus- 
ters that yielded single units (top half of each plate, on the right), 
a brief sample of continuous recordings from each channel 
(digital oscilloscope, bottom half on the right), and statistics for 
cluster separation. Distributions for these statistics are shown in 
Fig. 3 B’-E’. J3 depicts the ratio of between-cluster to within- 
cluster scatter; high values signify good isolation. DB reflects the 
ratio of the sum of within-cluster scatter to between-cluster 
separation, and low values show good isolation. Labels in Fig. 3 
B',C’, and E’ identify each of the channels shown in Fig. 2 A—H. 

Fig. 2 demonstrates that clear, single-unit isolation from the 
macaque monkey cortex can be obtained with chronically im- 
planted microwire arrays. Fig. 2 A-G depicts a sample of channels 
in which at least two well-isolated single neurons were identified, 
30 days after the implantation surgery. These examples represent 
48% of all channels that yielded single units at this postsurgical time. 
Fig. 2.A-E depicts channels with high J3 and F statistics and low DB 
values, indicating the presence of well-separated and defined clus- 
ters. High J3 and F combined with low DB values were also 
observed in channels in which only one single neuron (Fig. 2 H and 
1) was identified in addition to multiunit activity. Fig. 2F depicts 
a channel whose cluster separation was near the mean of the J3 
distribution. The lower quartile of the J3 and F distributions are 
illustrated by the example in Fig. 2G. Interestingly, DB values for 
these two channels were slightly below average (see Fig. 3E’), 
suggesting a reasonable level of cluster separation. These examples 
underscore the relevance of considering multiple parameters for 
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assessing cluster separation. Another example of this requirement 
is illustrated in Fig. 2/. In this channel, a clear single unit (yellow 
cluster) was separated from multiunit activity (green cluster). 
Statistical characterization of cluster separation in this channel 
yielded an average J3 value (3.2). A high F value and a low DB 
suggested better cluster separation than implied by the J3 statistics. 

Fig. 2 A-D displays examples of channels in which clusters 
depicting single units were identified in the presence of multiunit 
activity. Inspection of the 3D PC plots and the continuous records 
in the digital oscilloscope show how single units were clearly 
separated from multiunit activity (red clusters in Fig. 2A, blue 
clusters in Fig. 2 B—D, and green clusters in Fig. 2 H and J). Channels 
like the ones presented in Fig. 2 A—I accounted for ~80% of our 
sample. Fig. 2/ displays a representative example of the remaining 
20%, channels in which a single unit was identified despite the fact 
that only one well-defined cluster was present in 3D PC space. 
Usually this occurred because the voltage threshold was set at a 
level higher than the background multiunit activity. These records 
were accepted as single units based on the existence of a well- 
defined cluster, regular and high-amplitude waveforms, and clear 
refractory periods in the ISI histograms. 

Fig. 34 depicts the Vpp distribution of all single neurons 
isolated in monkeys 1 and 3, 30 days postsurgery. The mean + 
SE value of this distribution (115 + 3.1 ~V) corresponded to a 
signal-to-noise ratio of ~5.5:1. Notice that units as large as 200 
pV were obtained. Fig. 3 B-E depicts the distributions of F, J3, 
PsF, and DB statistics calculated for channels in which no single 
units were isolated (control computations; see Methods) in 2D 
PC space. The mean (+SE) values for these channels were: F 
(2D) = 2.3 + 0.1, J3 (2D) = 0.7 + 0.02, PsF (2D) = 1,906.5 + 
314.1, and DB (2D) = 0.49 + 0.01. For channels with single units 
(Fig. 3 B'-E') the means were: F (2D) = 11.5 + 1.0, J3 (2D) = 
3.8 + 0.4, PsF (2D) = 37,364.8 + 6,026.9, and DB (2D) = 0.32 + 
0.01. For each of these four parameters, the distributions ob- 
tained for channels with single units differed significantly from 
channels without single units (¢ test for independent samples, 
P <1 X 10-7; Mann-Whitney U test, P < 1.0 x 1077; 
Kolmogorov-Smirnov two-sample test, P < 0.001). 

Further quantification revealed that 56 of 96 implanted mi- 
crowires (58%) in monkey 1 yielded at least one single neuron 
30 days after the surgery. This included 25 of 32 (78%) microw- 
ires in right PMd, 13 of 16 (81%) in right M1, 11 of 16 (69%) in 
right S1, and 7 of 16 (69%) in left M1. The overall mean of 
neurons isolated per microwire implanted was 0.9 (83 neurons 
from 96 microwires). Divided by cortical area, we recorded 1.3, 
1.4, 0.8, and 0.4 neurons per microwire in right PMd, right M1, 
right S1, and left M1, respectively. When only the microwires 
containing at least one single unit were considered (56 microw- 
ires), these values changed to 1.5 overall, and by area: PMD = 
1.6, right M1 = 1.7, right S1 = 1.2, and left M1 = 1.0. 

The largest sample of neurons recorded in a single session was 
obtained in monkey 3. Using a 384-channel MAP cluster, we 
were able to record the extracellular activity of 247 single cortical 
neurons from 384 microwires (see Fig. 5B, mean = 0.6 neurons 
per implanted microwire), 30 days after the implantation surgery 
(Fig. 4). The mean + SE Vpp of the waveforms displayed in Fig. 
4 was 126 + 4.2 wV, which corresponded to a mean signal-to- 
noise ratio of ~6:1. Single units with Vpp as high as 300-500 wV 
were present in monkey 3. The mean cluster separation statistics 
for the recording session depicted in Fig. 4 were: F (2D) = 11.1 + 
1.2, J3 (2D) = 3.4 + 0.4, PsF (2D) = 35,471 + 7,401, and DB 
(2) = 0.34 + 0.02. Single units were recorded from the left M1 
(87 neurons from 96 microwires, Al-F7), left PMd (53 neurons 
from 96 microwires, F8-H7 and J3-K7), left S1 (27 neurons from 
64 microwires, H8-J2), the right S1 (52 from 96 microwires, 
K8-N11), and the right M1 (28 neurons from 32 micro- 
wires, N12-P7). In this recording session, the percentage of 
microwires yielding at least one single neuron was 35% (135 
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Fig. 3. (A) Vpp distribution from 330 single neurons 

recorded in two monkeys, 30 days after the surgical 
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values for channels without single units. (C’) Distribu- 
tion of J3 (2D) values for channels with single units. (D) 
Distribution of PsF (2D) values for channels without 
single units. (D’) Distribution of PsF (2D) values for 
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of 384 microwires). This percentage was higher in the M1 (51%, | implanted microwire) was higher than those obtained for PMd 
65 of 128 microwires) than in the PMd (26%, 25 of 96 microw- _—_ (0.6) and S1 (0.5). When only microwires containing single units 
ires) and the S1 (28%, 45 of 160 microwires). The mean of — were considered (135 microwires), the overall mean rose to 1.8 
neurons isolated per implanted microwire varied significantly neurons per microwire. When this calculation was broken down 
according to cortical area. Again, the M1 mean (0.9 neurons per _ according to cortical area, recordings in PMd yielded 2.1 neu- 
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Fig. 4. Samples of action potentials from 247 single neurons recorded in 
monkey 3, 30 days after implantation. Sample includes left M1 (A1-F7), left 
PMd (F8-H7 and J3-K7), left $1 (H8-J2), right $1 (K8-N11), and right M1 
(N12-P7). Numbers next to the sampled waveforms display their Vpp (V). 
Mean + SE J3 (2D), F (2D), and DB (2D) statistics for the sample are plotted at 
the bottom right. 


rons per viable microwire; M1 (1.8) and S1 recordings (1.8) 
produced slightly smaller values. 

By combining results from the three individual recording 
sessions, we verify that 3-4 weeks after the implantation surgery 
a total of 421 neurons were recorded from 576 implanted 
microwires (a mean of 0.7 neurons per implanted microwire) in 
the three monkeys. On average, 54% of the implanted microw- 
ires in the three monkeys yielded at least one isolated cell. Thus, 
if one considers only the total number of viable microwires in 
these recording sessions, the overall mean of neurons discrim- 
inated per viable microwire was 1.6. 


Short- and Long-Term Stability of Single-Unit Isolation. In addition to 
providing a very reasonable yield, chronic microwire recordings 
were stable at both short- and long-term time scales (see Figs. 7 and 
8, which are published as supporting information on the PNAS web 
site). Quantitative analysis revealed that PC clustering depicting 
single-unit isolation remained stable throughout recording sessions 
that lasted for 1-2 h. This finding was confirmed by the fact that 
neither the ISI histograms calculated for these single units nor the 
cluster separation statistics calculated for these channels varied 
significantly during the recording sessions. 

Fig. 5A displays waveform samples from 58 simultaneously 
recorded neurons in monkey 1, 18 months after the initial 
implantation surgery. The Vpp distribution (112.6 + 8.9 pV) for 
the action potentials of these neurons is presented in Fig. 34’. No 
statistical difference was found between this Vpp distribution 
and that obtained from data collected 30 days postsurgery. 
Means + SE of three cluster separation statistics for this 
recording session are displayed at the bottom of Fig. 5A. The 
obtained values [F (2D) = 11.9 + 1.2, J3 (2D) = 4.6 + 1.0, and 
DB (2D) = 0.3 + 0.02] did not differ from those obtained 30 days 
postsurgery. These 58 neurons were discriminated from 34 viable 
microwires, which corresponded to 35% of the 96 microwires 
originally implanted. The number of neurons isolated per im- 
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Fig. 5. (A) Waveform sample from 58 single neurons recorded 18 months 
after the implantation surgery. Numbers next to waveforms depict the mean 
Vpp (1V) of the action potentials. Right PMd (A1-B2), right M1 (B3-E6), right 
S1 (E7-F7), and right SMA (F8-H2). Mean = SE J3 (2D), F (2D), and DB (2D) 
statistics for the sample are plotted at the bottom right. (B-G) Single neuron 
isolation quality is illustrated as described in Fig. 3. (B) Four single units 
discriminated from the multiunit activity (blue). (C) Clear isolation of a single 
neuron (yellow) from the multiunit activity (green) is confirmed by high J3 and 
F values and a low DB. (D) Channel with a clear single unit, but with a smaller 
signal-to-noise ratio than C. ISI histograms show a clear refractory period. (E) 
Two clearly distinct single units produced clusters with average J3, close to 
average F, and low DB. (F) Two units (green and yellow) discriminated from the 
multiunit activity (blue) yielding high J3, F, and low DB statistics. Distinct ISIs, 
waveform shapes, and amplitudes further supported their designation as 
single units. (G) Channel with average J3 and slightly lower than average F and 
DB statistics. Clear differences in ISI histograms and waveform shape and 
amplitude confirm nice separation of units (green and yellow) from multiunit 
activity (blue). ISI histogram time scales are 10 ms (B-F) and 100 ms (G). 
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planted microwire in different cortical areas was the following: 
PMd = 0.3, M1 = 1.8, S1 = 0.8, and SMA = 0.7, total = 0.6. 
When only the number of viable microwires was considered, the 
neuronal yield was: PMd = 1.7 neurons per microwire, M1 = 1.9, 
S1 = 1.3, and SMA = 1.8. The mean (1.7 neurons per viable 
microwire) was the same as the value obtained for monkeys 1 and 
3, 30 days postsurgery. As in recordings obtained 30 days 
postsurgery (Fig. 3), Fig. 5 B-G demonstrates that 18 months 
after implantation, recordings from single microwires allowed 
multiple single units to be clearly separated from background 
multiunit activity (blue clusters in Fig. 5 B and F-G, and green 
clusters in Fig. 5 C and D). 


Discussion 

This article describes a paradigm for obtaining long-term, mul- 
tielectrode recordings in macaque monkeys. In each of three 
rhesus monkeys, simultaneous recordings were obtained from 
83-247 neurons, distributed across five cortical fields. These 
neuronal samples, which were each obtained in a single day, are 
comparable to those obtained with single-electrode methods in 
weeks of recording. Long-term experiments in one subject 
revealed that high-quality single-unit recordings could be ob- 
tained for 18 months. 


Methodology. The principal issue addressed in this study was 
whether the methods we have previously applied to rodents and 
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New World primates, which have few and shallow sulci, can be 
applied to macaques, with their larger brains, deeper sulci, 
greater brain pulsations, and lower cell-packing densities. Our 
findings suggest that the methods used here are robust and 
reliable in macaque monkeys. Indeed, the yield of single neurons 
in the present study exceeds dramatically those reported with 
previous methods (9-14). In addition, we have shown the 
capability to record simultaneously from many cortical areas in 
both hemispheres for many months. 

Various factors contributed to the robustness and reliability of 
the present method. First, high-quality chronic recordings were 
most reliably obtained (i) when the surgical procedure involves 
very slow implantation of the microwire arrays and (ii) when 
single-unit recordings observed during the implantation proce- 
dure were of high quality. This slow implantation procedure 
differs dramatically from that used for the Utah array, for 
example, which enters the cortex at high speed (9, 10). Another 
important factor seems to be the configuration of the electrode 
tip. We found that, like in other species, blunt-tip microwires 
work well in a variety of cortical areas in macaques. 

Not only do the methods described here work well in ma- 
caques, in some ways they exceed previous results in other 
species. For example, the 247 neurons isolated from one subject 
represent a 2-fold increase in yield compared with our previous 
studies in rats (15) and New World monkeys (7, 8). This 
improvement results mainly from the ability to implant a larger 
number of electrodes in macaque brains. Furthermore, our 
previous multielectrode recordings in rats have usually involved 
arrays of 16 microwires. In the present study, we used arrays of 
32-128 microwires. Thus, one can isolate neurons from up to 40 
mm? of cortex per implant, which is more than a third of the 
forelimb representation in the macaque M1 (19). 


Single-Unit Sorting and Statistics. The larger number of electrodes 
and the presence of more than one single unit on many of those 
electrodes presents some challenges. With traditional neuro- 
physiological methods, subjective judgment is used to decide 
whether the records consist of single units. The present use of PC 
analysis is only one among many methods for sorting spikes from 
multiunit noise and from each other. PC analysis appears to be 
very useful, when used in conjunction with other criteria. 
Similarly, the precise statistical methods used to validate single- 
unit isolation have not yet been standardized in neurophysiology. 
There are probably alternatives to the four statistics used here, 
which would work as well or better. Yet, no one statistic seems 
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to provide a unique solution to this problem. Instead, we propose 
that using several different statistics in conjunction with other 
criteria for cell isolation (see Methods), showing the distribution 
of those statistics for channels with and without single units (Fig. 
3), and illustrating examples that span the entire range of single 
neuron isolated in a given study (see Fig. 3) should be used to 
document the quality of multielectrode recordings. 


Potential Applications. This methodology should be of value 
whenever a large sample size is important for testing a hypoth- 
esis. For example, simultaneously recording neuronal popula- 
tions on the order of hundreds of cells should enable better tests 
of ideas about the role of precise timing and large-scale neuronal 
interactions in cortical computations (20-24). The stability of 
single-unit recording during a single recording session will 
enable investigators to study a larger set of independent vari- 
ables. And by inserting cannulae in the arrays, pharmacological 
methods for inactivation or stimulation (25) could be combined 
with recordings. Moreover, the present methods should result in 
an improved ability to study neuronal activity during learning. A 
modest beginning has been made with single-electrode methods 
(e.g., ref. 26), but such studies required extraordinarily rapid 
learning and yielded relatively small neuronal samples. At a 
minimum, it should be possible to study anything that can be 
learned within a few days or even weeks. 

Finally, the methods described here could have an impact on 
testing cortical neuroprosthetic devices (8, 13, 27). The main 
long-term goal behind this research is to allow neurological 
patients to use spared brain tissue to restore function. To this 
end, we have trained owl monkeys to use their own cortical 
activity to control either the movements of a robot arm or a 
cursor (8). Extension of this experimental design should also 
allow real-time implementations of computational models that 
predict the animal’s perceptual, motor, or cognitive capabilities 
from brain activity and allow these predictions to be tested in real 
time, before an animal’s response (28). 


This article is dedicated to Dr. Cesar Timo-Iaria for his >50 years of 
continuous contributions to neurophysiology. We thank Drs. P. Lange, 
J. Harer, and R. S. Williams for their support and E. Phelps, L. Oliveira, 
S. Halkiotis, L. Spence, D. Schmid, and S. Wiebe for outstanding 
technical assistance. This work was supported by the National Institutes 
of Health, the National Science Foundation, the Defense Advanced 
Research Projects Agency, the James S. McDonnell Foundation, and a 
Duke University instrumentation award (to M.A.L.N.). 


14. Hoffman, K. L. & McNaughton, B. L. (2002) Science 297, 2070-2073. 

15. Nicolelis, M. A. L., Ghazanfar, A. A., Faggin, B. M., Votaw, S. & Oliveira, 

L. M. O. (1997) Neuron 18, 529-537. 

16. Wheeler, B. C. (1999) in Methods for Neural Ensemble Recordings, ed. Nicolelis, 

M. A. L. (CRC, Boca Raton, FL), pp. 61-77. 

7. Spath, H. (1980) Cluster Analysis Algorithms (Ellis Horwood, New York). 

18. Davies, D. L. & Bouldin, D. W. (1979) IEEE Trans. Pattern Anal. Machine 

Intell. 1, 224-227. 

19. Woolsey, C. N., Settlage, P., Meyer, D. R., Sencer, W., Hamuy, T. P. & Travis, 
A. M. (1952) Res. Publ. Assoc. Res. Nerv. Ment. Dis. 30, 238-264. 

20. Singer, W. & Gray C. M. (1995) Annu. Rev. Neurosci. 18, 555-586. 

21. Vaadia, E., Haalman, 1, Abeles, M., Bergman, H., Prut, Y. Slovin, H. & 
Aertsen, A. (1995) Nature 373, 515-518. 

22. Newsome, W. T., Shadlen, M. N., Zohary, E., Britten, K. H. & Movshon, J. A. 
(1995) in The Cognitive Neurosciences, ed. Gazzaniga, M. S. (MIT Press, 
Cambridge, MA), pp. 401-414. 

23. Ahissar, E. & Arieli, A. (2001) Neuron 32, 185-201. 

24. Nicolelis, M. A. L. & Fanselow, E. E. (2002) Nat. Neurosci. 5, 517-523. 

25. Krupa, D. J., Ghazanfar, A. A. & Nicolelis, M. A. L. (1999) Proc. Natl. Acad. 
Sci. USA 96, 8200-8205. 

26. Mitz, A. R., Godschalk, M. & Wise, S. P. (1991) J. Neurosci. 1, 1855-1872. 

27. Nicolelis, M. A. L. (2001) Nature 409, 403-407. 

28. Nicolelis, M. A. L. & Ribeiro, S. (2002) Curr. Opin. Neurobiol. 12, 

602-606. 


Nicolelis et al. 


124 


JOURNAL OF 
NEUROSCIENCE 
METHODS 


www.elsevier.com/locate/jneumeth 


ELSEVIER 


Journal of Neuroscience Methods xxx (2003) xxx—xxx 


A low power multichannel analog front end for portable 
neural signal recordings 


Iyad Obeid**, Miguel A.L. Nicolelis®>-4, Patrick D. Wolf@ 


* Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA 
> Department of Neurobiology, Duke University, Durham, NC 27708, USA 
© Department of Psychological and Brain Sciences, Duke University, Durham, NC 27708, USA 
4 Co-director-Duke Center ‘for Neuroengineering, Duke University, Durham, NC 27708, USA 


Received 21 February 2003; received in revised form 9 June 2003; accepted 12 September 2003 


Abstract 


We present the design and testing of a 16-channel analog amplifier for processing neural signals. Each channel has the following features: (1) 
variable gain (70-94 dB), (2) four high pass Bessel filter poles (f_3ag = 445 Hz), (3) five low pass Bessel filter poles (f_3u53 = 6.6 kHz), and 
(4) differential amplification with a user selectable reference channel to reject common mode background biological noise. Processed signals 
are time division multiplexed and sampled by an on-board 12-bit analog to digital converter at up to 62.5k samples/s per channel. The board 
is powered by two low dropout voltage regulators which may be supplied by a single battery. The board measures 8.1 cm x 9.9cm, weighs 
50 g, and consumes up to 130 mW. Its low input-referred noise (1.0 WVpms) makes it possible to process low amplitude neural signals; the 
board was successfully tested in vivo to process cortically derived extracellular action potentials in primates. Signals processed by this board 
were compared to those generated by a commercially available system and were found to be nearly identical. Background noise generated by 
mastication was substantially attenuated by the selectable reference circuit. The described circuit is light weight and low power and is used 
as a component of a wearable multichannel neural telemetry system. 


© 2003 Elsevier B.V. All rights reserved. 


Keywords: Single unit recording; Neural signal processing; Low-power 


1. Introduction 


Multichannel neural recordings are used to study the 
behavior of populations of neurons in various parts of the 
brain (Hampson et al., 1999; Katz et al., 2002; Ohl et al., 
2001; Wessberg et al., 2000). However, the requirement of a 
cable tether for acquiring data necessitates physical restraint 
of the subject. In order to study populations of neurons in 
freely moving subjects, we require a wearable multichannel 
neural telemetry system. This work presents the design and 
testing of a low-power 16-channel analog front end (AFE) 
for use in such a system. The AFE amplifies, filters, and 
digitizes neural signals, and has been tested in primates. All 
components are available off the shelf, and all production 
files for the required printed circuit boards have been made 
available online, thus making the circuit easily reproducible. 


* Corresponding author. Tel.: +1-919-660-5109; 
fax: +1-919-660-5405. 
E-mail address: io@duke.edu (I. Obeid). 


0165-0270/$ — see front matter © 2003 Elsevier B.V. All rights reserved. 


doi: 10.1016/j.jneumeth.2003.09.024 


2. System design 


The AFE is comprised of 16 parallel channels, each with 
four identical analog processing stages: preamplifier, differ- 
ential amplifier, filter, and variable gain (Fig. 1). Two 8:1 
multiplexers and three SPST switches provide reference se- 
lection for the differential amplifiers. A 16:1 time division 
multiplexer (TDM) selects channels to be sampled by the 
analog to digital converter (ADC). The gain is designed 
to vary between 3200 (70 dB) and 50,000 (94dB), and the 
bandpass filter corner frequencies are placed at 350 and 
8 kHz. The front end is designed to be driven by an active 
headstage. The schematic for a single channel is shown in 
Fig. 2. 


2.1. Analog block 
2.1.1. Preamplifier 


The preamplifier is comprised of the unity gain high-pass 
filter formed by C; and Rj, and the non-inverting low-pass 
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Fig. 1. The overall block diagram of the 16-channel analog front end. Electrode signals are input at the left. Referencing for the differential amplifiers 
is handled by a pair of 8:1 multiplexers and three switches. The user can therefore select either a bipolar (referenced to another channel) or unipolar 
(referenced to ground) reference for each of the two 8-channel groups. One channel can also be selected as the bipolar reference for the whole board. 
The variable gain for each channel is determined by a user controlled digital potentiometer. Processed signals are time-division multiplexed and sampled. 


The digitized data stream must be processed off-board. 


filter with gain formed by Ampl, R2, R3, and C2. The 
Maxim 4253 (Maxim IC, Sunnyvale, CA) op-amp used for 
Amp! features low noise density (7.9 nV//Hz), and may be 
shutdown to reduce power consumption when its channel is 
not in use. The preamplifier gain is fixed at 100. 


2.1.2. Differential amplifier 

The differential amplifier stage combines a gain of 10 
with a high-pass filter pole. Amp2 (Maxim 4199) provides 
the differential gain with a common mode rejection ratio 
(CMRR) of ~ 110dB, while the integrator in the feedback 


pathway adds the pole. Note that the differential amplifier’s 
Vet input is taken from one of the two 8:1 reference mul- 
tiplexers (Fig. 1). The 4199 may be shut down when not in 
use, and the feedback op-amp (OPA2244, Burr Brown/Texas 
Instruments, Dallas, TX) consumes only 50 pA. 


2.1.3. Reference selection 

A reference selection circuit allows the user to spec- 
ify the differential amplifiers’ reference signals. The AFE’s 
16-channels are divided into two groups of eight. The pream- 
plifier outputs for each group are wired into an 8:1 multi- 
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Fig. 2. Schematic for a full analog front end channel. The Vjn input is taken from a headstage buffered electrode, and the V;er input is taken from the 
output of the channel’s respective 8:1 multiplexer. Voy: connects to the 16:1 time division multiplexer. All components are available off-the-shelf. 
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plexer, allowing one of those channels to be chosen as the 
reference from that eight-channel group (Fig. 1). Addition- 
ally, three switches operated in tandem with the 8:1 multi- 
plexers allow for a single signal to be the reference for all 
16 channels, or for a separate ground electrode to be the ref- 
erence. The Analog Devices AD708 (Analog Devices, Nor- 
wood, MA) was selected for the two 8:1 multiplexers, and 
the Maxim 4626 was chosen for the SPST switches. 


2.1.4. Filters 

The filter stage is comprised of three Sallen-Key filters— 
one high-pass cascaded into two low-pass filters. We used 
these filters, in combination with those in the preamplifier 
and differential amplifier stages, to form Bessel filters. The 
Sallen-Key filter op-amp is the quad package Microchip 
MCP604. 


2.1.5. Variable gain 

The variable gain stage increases the system’s dynamic 
range, reducing the number of bits of resolution required in 
the analog to digital converter. The variable gain, controlled 
by a 200k&2 32-tap digital potentiometer (Maxim 5160), is 
evenly spaced between | and 16.5. The op-amp is the fourth 
device on the MCP604 used for the Sallen-Key filters. 


2.2. Digital block 


The analog signals are time-division multiplexed using the 
Analog Devices AD706 and sampled with the Analog De- 
vices AD7495. The AD7495 is a 12-bit low power (4.5 mW) 
successive approximation ADC with a maximum through- 
put rate of 1M samples/s, allowing each of the 16 channels 
to be sampled at up to 62.5k samples/s. Timing signals for 
the multiplexer and ADC must be generated off-board. 


2.3. Power supply 


Power for the board is supplied by two high precision, 
low dropout voltage regulators, Analog Devices REF191 and 
REF198, which generate 2.048 and 4.096 V, respectively. 
The 2.048 V rail is used as a virtual ground, effectively cre- 
ating a +2.048 V power supply without requiring an invert- 
ing voltage regulator or negative power supply. Since the 
REF191 (acting as the virtual ground) is incapable of sink- 
ing current, its output is buffered with a unity gain source 
follower (Analog Devices OP262). This virtual ground is 
used to ground the subject. The circuit is designed to be 
powered by an isolated power source, ideally a battery. 


2.4. Physical circuit 


A six-layer 8.1 cm x 9.9 cm printed circuit board (PCB) 
was designed for the 16 AFE channels, the reference selec- 
tion circuits, and the power supply, as well as two 30-pin 
digital I/O connectors and a 26-pin analog input connector. 
The analog channels were laid out in eight parallel rows on 


each side of the board. A high board density was achieved by 
using the smallest available hand-solderable parts (size 0402 
for the passive components), 6.25 mil signal trace widths, 
and 24 mil vias with 10 mil drill holes. The analog input con- 
nector includes the 16 input lines plus two power and two 
ground lines for powering an active headstage. The pream- 
plifier and differential amplifier stages were protected from 
electromagnetic radiation by a grounded metal shield placed 
over both sides of the board. 

The AFE requires 50 digital control signals that must be 
generated off-board. Power supply lines for the two voltage 
regulators must also be generated off-board. The digitized 
data serial bitstream is the only signal output by the AFE 
board. Digital I/O is handled through two parallel 30-pin 
connectors (Advanced Interconnect, West Warwick, RI). 
These carry the 50 control signals, plus four voltage refer- 
ence power supply lines, four ground lines, and the one dig- 
ital data output. The connectors may be used to control the 
AFE board using either a remote computer (via cable) or alo- 
cal digital board that plugs directly into the AFE. This board 
would house power supply circuitry and programmable 
logic to manage the AFE’s digital I/O (Obeid et al., 2004). 


3. Methods 


The AFE board was manufactured using standard PCB 
technology. Schematics, circuit board Gerber files, and a 
complete bill of materials may be found at http://dukebme. 
duke.edu/backpack. The AFE was characterized using a 
high precision evaluation deck (System-Two, Audio Pre- 
cision, Beaverton, OR), and tested in vivo on a macaque 
(Macaca mulatta). The subject had electrodes implanted in 
the somatosensory cortex near spontaneously firing neurons 
(Nicolelis et al., 1997). The electrodes were buffered with 
commercially available headstages (HST/8050-G20-GR, 
Plexon Inc., Dallas, TX) and connected via a short cable 
(FlatFlex, Parlex Inc., Methuen, MA) to the AFE’s input. 
Control signals for the AFE were provided from a custom 
designed digital board with programmable logic. 

The board’s in vivo performance was compared to that 
of a commercial neural data acquisition system (MNAP, 
Plexon Inc.). A “Y”-adapter was used to split the raw elec- 
trode signals. The first “Y” branch was buffered by a Plexon 
headstage, processed by the MNAP, and then saved using a 
digital tape deck (DA-38, TASCAM; 20 Hz—20kHz, F, = 
44.1kHz). The second branch was buffered with a custom 
headstage (Obeid et al., 2003), processed by our AFE (Fs = 
30 kHz), and stored as eight-bit samples on a PC hard drive. 
The signals were compared by resampling the AFE signals 
at 44.1 kHz in MATLAB and then normalizing both signals. 

The selectable reference channel feature was evaluated 
by recording signals during mastication. Signals were first 
recorded from two adjacent microwire electrodes (channels 
1 and 2) referenced to a separate ground while the sub- 
ject chewed continuously. Channel 1 was then referenced to 
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channel 2 and recorded during a similar period of continu- 
ous chewing. The recordings were analyzed for evidence of 
signal artifacts generated by the chewing. 


4. Results 


The AFE board measures 8.1cm x 9.9cm and weighs 
50 g, including all sockets, components, and the lightweight 
metal shielding. The AFE’s performance is summarized in 
Table 1. Predicted values are based on SPICE simulations 
and component data sheets. Differences between the mea- 
sured and predicted values of the gains and pole locations 
can be attributed entirely to capacitor and resistor toler- 
ances. The CMRR value of 42.3 dB is the mean (n = 15) of 
the CMRR measurements made at 1 kHz between adjacent 
channels on the board. The measured input referred noise 
value corresponds to a noise density of 12.8 nV/./Hz. Noise 
measurements were made under normal laboratory condi- 
tions (i.e. normal ambient noise sources) without an active 
headstage. The maximum signal to noise plus distortion ra- 
tio (SNDR) ranged from 29.4dB at the maximum gain to 
52.4 dB at the minimum gain. 

Fig. 3 shows a comparison of neural signals processed 
in parallel by the AFE and the Plexon MNAP; the signals 
are nearly identical. The average root mean square error be- 
tween the AFE and MNAP signals is 5.2 wVpms. This is at- 
tributable to (1) AFE circuit noise, (2) noise differences be- 
tween the Plexon and custom headstages, (3) differences in 
filtering between the AFE, the MNAP, and the anti-aliasing 
filter of the digital tape recorder, and (4) the resampling re- 
quired to compare the signals. 


Amplitude (uVolits) 
o 


1 2 3 
Time (msec) 


Table 1 


Summary of electrical characteristics 


Parameter Measured value Predicted value 
Gain (low) 69.4 + 0.11 dB 70.0 dB 
Gain (high) 93.4 + 0.08 dB 94.4 dB 
f-3ap (low) 445 + 6.5 Hz 390 Hz 
f-3ap (high) 6.55 + 0.5 kHz 7.59 kHz 
CMRR 42.3 dB 39.2 dB 
(worst case) 

Group delay (maximum)  2.0ms 3.8 ms 
Noise (input referred) 1.0 »VRus 0.7 w»VRus 
Power consumption 130 mW 130 mW 
Full scale input range 1.4mV at minimum gain 1.3mV 

87 pV at maximum gain 78 pV 


Common mode range >+10.5V 


The utility of the selectable reference electrode matrix is 
demonstrated if Fig. 4. The top two traces, recorded simul- 
taneously from two adjacent electrodes during mastication, 
demonstrate that large signal artifacts are common to both 
electrodes. In the third trace, the reference selection circuit 
is used to make a bipolar recording between the electrodes 
of the first two traces. No signal artifacts are present. 


5. Discussion 


Our design will be useful for any research requiring a 
low power multichannel circuit for processing and digitiz- 
ing extracellular neural signals. Aspects of this architecture 
such as the ground referencing scheme or the low noise/high 
gain preamplifiers may be adapted for related applications. 


Fig. 3. Equivalence of the analog front end to a commercial recording system. Samples of primate signals recorded in parallel through our analog front 
end (dashed traces) and the Plexon MNAP system (solid traces). The signals are nearly identical. The mean root mean square error between the two sets 


of waveforms is 5.2 1Vpms.- 
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Fig. 4. Use of the reference channel selection feature to remove signal artifacts. Traces (a) and (b) were recorded simultaneously from adjacent microwire 
electrodes during mastication using unipolar ground referencing. Signal artifacts are common to both traces. In (c), a bipolar recording between the 


electrodes of (a) and (b) attenuates mastication noise. 


Reproducing our design requires no special in-house facil- 
ities; all circuit components are available off-the-shelf and 
are hand solderable, and the printed circuit board construc- 
tion may be outsourced. 


5.1. Design considerations 


The small amplitude extracellular signals typically en- 
countered in neural recordings necessitate a low-noise ap- 
proach. Since noise performance is always limited by the 
first stage, the preamplifier is designed to be the quietest and 
highest gain stage. Preamplifier noise is limited by combin- 
ing a low-noise, high power op-amp and a low-pass filter. 
The non-inverting architecture prevents loading of the the 
high-pass filter. While lower noise amplifiers are available, 
they generally require more power; our preamplifier design 
is a trade-off between power consumption and noise. 

The differential stage is placed immediately after the 
preamplifier to (1) apply more gain in an early stage and 
(2) improve signal quality by allowing for the rejection of 
common mode background signals. We have shown that ref- 
erence selection can be instrumental in attenuating common 
mode signal artifacts such as those generated by motion 
or chewing. The active filter in the differential amplifier’s 
feedback pathway improves the common mode range by 
adding a high-pass pole to attenuate low frequency offset 
voltages. 


To minimize waveform distortion, the AFE is designed 
with Bessel filters. This preserves the fidelity of the signals, 
a necessity if any waveform-based spike sorter is used. Since 
Bessel filters minimize phase distortion at the expense of 
sharp filter roll-offs, higher order filters are used. The sys- 
tem has a total of four high-pass and five low-pass poles. 
The measured phase response of our AFE is approximately 
linear (r7 = 0.98) making the group delay (—d@/d f) nearly 
constant in the passband. The maximum group delay at any 
frequency is ~2 ms. 

The variable gain stage is placed after the three Sallen-Key 
filters in order to drive the capacitative input load of the 16:1 
time-division multiplexer. The system’s overall resolution 
varies from 18.6nV/bit at maximum gain and resolution to 
4.9V/bit at minimum gain and resolution. 

Although we use a precision differential amplifier with 
a CMRR of 110dB, we were unable, in practice, to mea- 
sure CMRRs greater than ~42 dB. This can be accounted 
for by the device tolerances in the preamplifier stage; using 
+0.1% resistors and 5% capacitors in the preamplifier, the 
expected worst case CMRR at | kHz is 39.2dB. The 8:1 
multiplexers providing the reference signals for the differ- 
ential amplifiers were found to have a negligible effect on 
the CMRR, even when a single multiplexer was driving all 
16 reference lines. To verify our calculations, the pream- 
plifier was rebuilt using +1% resistors and +10% capaci- 
tors. In this case, we measured the CMRR to be 33.3 dB at 
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1 kHz, which agrees with the predicted worst case CMRR 
of 25.4 dB. 

In order to facilitate a portable, battery powered product, 
an emphasis was placed on minimizing power consumption. 
Low power op-amps with shutdown capability were used 
where possible. The predicted power consumption for each 
active channel is 7.3mW. The analog to digital converter 
(AD7495) and AC ground buffer (OP262), which are always 
on, consume a combined 10.2 mW. The two low dropout 
voltage regulators together only use 0.38 mW. 


5.1.1. Noise 

The predicted input referred noise for the preamplifier 
stage is 0.7Vpms, which includes the contributions of the 
opamp (Maxim 4253) and the passive feedback components. 
The prediction increases by less than 0.5% when the con- 
tribution of differential amplifier stage is added. The actual 
measured input referred noise (1.01Vpms) reflects a sum 
of the circuit noise and the ambient electromagnetic noise. 
This increase over the predicted noise value was measured 
in spite of the grounded metal shields. The noise sources 
reduce the overall system resolution; although the analog to 
digital converter produces 12 bit samples, only 10.3 bits are 
more significant than the noise floor at the minimum gain 
and 6.3 bits at the maximum gain. 


6. Conclusion 
This work has presented the design and test results of a 


low power analog front end board for multichannel neural 
signal processing. This board was designed as part of an 


ongoing research effort to develop a portable wireless neural 
signal recording system. 
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Abstract 


We present the design, testing, and evaluation of a 16 channel wearable telemetry system to facilitate multichannel single unit recordings 
from freely moving test subjects. Our design is comprised of (1) a 16-channel analog front end board to condition and sample signals derived 
from implanted neural electrodes, (2) a digital board for processing and buffering the digitized waveforms, and (3) an index-card sized 486 
PC equipped with an IEEE 802.11b wireless ethernet card. Digitized data (up to 12 bits of resolution at 31.25k samples/s per channel) is 
transferred to the PC and sent to a nearby host computer on a wireless local area network. Up to 12 of the 16 channels were transmitted 
simultaneously for sustained periods at a range of 9 m. The device measures 5.1 cm x 8.1cm x 12.4cm, weighs 235 g, and is powered from 
rechargeable lithium ion batteries with a lifespan of 45 min at maximum transmission power. The device was successfully used to record 


signals from awake, chronically implanted macaque and owl monkeys. 


© 2003 Elsevier B.V. All rights reserved. 


Keywords: Neural telemetry; Single unit recording; Wireless; Neural data acquisition; Portable 


1. Introduction 


Multichannel single unit recordings have been used to 
demonstrate the feasibility of building brain machine inter- 
faces. By correlating subject movement with neural activity, 
researchers have been able to control primitive prosthetic 
limbs (Taylor et al., 2002; Wessberg et al., 2000). Progress 
in this area is limited by the requirement that subjects be 
restrained and tethered to the neural recording hardware 
(Nicolelis, 2001). Although wireless neural recording de- 
vices have been reported (Akin et al., 1998; Irazoqui-Pastor, 
2002; Maki, 1998; Mojarradi, 2003), none are sufficient for 
recording multiple channels of single units in freely mov- 
ing subjects. At present, inductive coil based systems either 
support too few channels or have too small of a transmission 
range. Analog systems are susceptible to transmission chan- 
nel noise, which may adversely affect spike sorting (Kim 
and Kim, 2000). Furthermore, future on-board processing, 
such as spike detection and spike sorting may require al- 
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gorithms better suited to the digital domain (Chandra and 
Optican, 1997; Hulata et al., 2002; Letelier and Weber, 
2000). 

We present a design for a wearable telemetry system 
weighing 235 g capable of transmitting up to 12 of 16 sig- 
nals from implanted neural electrodes. Signals are digitized 
at 31.25k samples/s per channel at eight, ten, or 12 bits of 
resolution, and telemetered using an IEEE 802.11b wireless 
ethernet card to a host computer up to 9m away. The device 
has been built, characterized, and tested in vivo. 


2. System design 
2.1. Overview 


Our system consists of a wearable telemetry unit which 
communicates with a stationary host computer. The teleme- 
try unit has four main parts: (1) the analog front end (AFE), 
(2) the digital board, (3) the wearable computer, and (4) 
the battery. The AFE (described in detail elsewhere ((Obeid 
et al., 2004)) conditions and time-division multiplexes up to 
16 neural signals before sampling them at 31.25 k samples/s 
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per channel. Fifty digital inputs are required to control chan- 
nel enables, variable gain amplifiers, and analog to digital 
converter timing. These controls are generated on the digi- 
tal board, which also provides regulated power, frames the 
digitized data, and communicates with the wearable com- 
puter. The wearable computer acquires the data from the 
digital board, transmits it to the host computer via a IEEE 
802.11 wireless network, and receives control commands 
from the user. The telemetry system is powered from a pair 
of rechargeable lithium-ion batteries. The functionality of 
the various parts of the telemetry system are explained be- 
low in more detail. 


2.2. Digital board 


The digital board consists of three main components: 
(1) power regulation circuitry, (2) a complex programmable 
logic device (CPLD), and (3) a first-in first-out (FIFO) mem- 
ory. The CPLD (Fig. 1) generates both static and timing 
control signals. The static signals are set by the user, and 
are loaded into registers in the CPLD. These lines directly 
assert control signals on the AFE board. The timing signals 
coordinate the operation of the time-division multiplexer, 


| toAFE | CPLD 


the analog to digital converter (ADC), and the FIFO. Time 
is divided into sets of 24 periods of a 12 MHz clock, since 
24 periods is the time it takes to sample and convert one 
channel. A priority encoder consults a channel enable list 
and determines whether to enable or disable the ADC for 
the current channel; if the ADC is disabled, the CPLD idles 
until the end of the 24 clock-cycle set before continuing to 
the next channel. Since each set takes 24/12 MHzs and each 
channel is processed once every 16 sets, the effective sam- 
pling rate is 1/[(24 x 16)/12 MHz] = 31.25 k samples/s per 
channel, or one sample per 32 us. 

The digitized data is passed serially from the ADC to 
the CPLD, where it is organized into bytes and stored in 
the FIFO. In order to reduce the data bandwidth, the user 
may opt to drop either two or four bits of the 12 gener- 
ated by the ADC. In these cases, the eight most signifi- 
cant bits are written to the FIFO while the remaining bits 
are stored in a temporary register; when this register ac- 
cumulates eight bits, it is moved to the FIFO and then 
cleared. Up to 24ms of data are stored in the FIFO before 
being transferred to the wearable computer’s parallel port; 
a handshake timing module in the CPLD negotiates this 
transfer. 


1 i * 8:1 Mux (A) Select & Enable (4) 
* 8:1 Mux (B) Select & Enable (4) 


* Ground Select Switches (3) 


17, } * Variable Gain Clock (16) 


* Variable Gain Up/Down (1) 


ura * Channel Enable (16) 


Encoder 


2, | |* ADC Chip Select (1) 
T1* ADC Clock (1) 


1/ i »/* ADC Bit-Serial Input (1) 


Fig. 1. Control signals on the digital board are generated by a CPLD. The CPLD generates timing signals for the AFE’s time division multiplexer and 
analog to digital converter. It also has registers for static control signals, such as channel enables and ground select settings. The CPLD receives user 


commands via the wearable PC. 


* Conversion Resolution (2) 
ae 
Priority 


* Bus Address (4) 


Demuttiplexer. CPU 
(486, DOS) 
wi Wireless 


LAN PC Card 


Handshake 
Timing 
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2.3. Wearable computer 


Telemetry is handled by a wearable computer (1.9cm x 
7.3cm x 9.2cm, 77 g, 486 BASE/CORE, Compulab, Haifa, 
Israel) fitted with an IEEE 802.11b wireless ethernet PCM- 
CIA card (PC4800, Cisco Systems, San Jose, CA). The 
wearable computer, based on a 66 MHz 486 AMD proces- 
sor, runs custom software under DOS to control data acqui- 
sition and wireless communication. A pair of custom-made 
6in. RF cables are used to offset the antenna of wireless 
ethernet card, reducing the proximity of the radiated elec- 
tromagnetic power (100 mW) from the AFE. The host com- 
puter can accept the data through a wireless access point or 
through another radio card. 

During a typical recording session, two network sockets 
are opened between the host and the wearable computer. 
A TCP/IP socket is used to send user commands from the 
host to the wearable, while a UDP socket transmits the data. 
Because it has less overhead than TCP/IP and lacks error 
checking, the UDP protocol is better suited for high band- 
width data transmissions. The wearable PC inserts an eight 
bit counter into each data packet to detect lost data, although 
there is no means for recovery or retransmission. 


2.4. Power supply 


The system is powered from a pair of rechargeable lithium 
ion batteries (UBC383562, Ultralife). They have a capacity 
of 1200 mAh at an average of 3.7 V and measure 7.6mm x 
35mm x 62mm. The battery plugs into the digital board, 
and a pair of DC to DC converters (MAX1701, Maxim 
IC, Sunnyvale, CA) produce power supply voltages of 3.3 
and 5.0 V. The 3.3 V supply powers the FIFO and CPLD, 
and both supplies are used to power the wearable computer 
and the AFE. Table | shows the overall power utilization 
for the telemetry system. The expected battery lifespan is 
(1200 mAh x 3.7 V)/4.0 W, or approximately 67 min. 


2.5. User interface 


The user communicates with the telemetry system through 
a host computer with a wireless interface. A custom software 
package allows the user to specify the active channels, the 


Table 1 
Breakdown of the predicted noise consumption for each of the major 
components of the wearable telemetry system 


Component Power consumption 
AFE 130 mW 

Wearable PC 1.31 W 

Radio card 2.45 W 

CPLD 29 mW 

FIFO 116mW 

Total 4.0W 


The wearable PC and wireless radio card consume 94% of the power. 


Table 2 
Number of bits exceeding the noise floor (worst case) 


Variable Number of bits exceeding Number of bits exceeding 
gain (0-31) noise (predicted) noise (measured) 
0 8.8 9.4 
7 6.7 8.1 
15 a7. 7.2 
23 5,2 6.7 
31 4.8 6.3 


gain setting for each channel, the reference channels, and 
the ADC resolution. These preferences are transmitted to 
the wearable computer using the TCP/IP protocol, and are 
relayed to the CPLD where they are stored in the appropriate 
registers. The user also uses this interface to specify how the 
received data is stored and displayed on the host computer. 


3. Results 


The wearable telemetry unit was built and tested. The 
bill of materials, schematics, and software are available, 
http://dukebme.duke.edu/backpack. The digital board mea- 
sured 5.6cm x 8.1cm. The completed telemetry unit, 
including battery, measured 5.1cm x 8.lcm x 12.4cm 
and weighed 235 g. Without the battery, the unit measured 
3.5cm x 8.1cm x 12.4cm and weighed 195 g. 

The system achieved sustained data transfers of 12 of the 
16 input channels in the eight-bit resolution mode. The max- 
imum number of channels at ten and 12 bits of resolution 
was nine and eight, respectively. The lifespan of the bat- 
tery was 45 min at the maximum data rate, and the range 
was 9m. The range was measured in a normal laboratory 
setting with no precautions taken to limit multipath reflec- 
tions or electromagnetic interference. The latency through 
the telemetry system from the input of the AFE to the input 
of the FIFO was 680 ws. 

The number of bits exceeding the background noise level 
(Table 2) was found to exceed the worst case predictions 
based on the input referred noise of the AFE (1.1 wV,,,,). 
Since the measured values exceed the predictions based on 
the AFE noise, it is inferred that no significant digital switch- 
ing noise from the digital board is added to the neural data. 

The device was tested in vivo with one rhesus macaque 
(Macaca mulatta) and one owl monkey (Aotus nancymae), 
both with multiple cortical implants of microwire bundles. 
The subjects were confined in a restraining chair during 
the tests. The telemetry unit was attached to the restrain- 
ing chair and connected to the implanted electrodes using 
a commercial headstage (HST/8050-G20-GR, Plexon Inc., 
Dallas, TX) and a short flexible cable (Parlex, Methuen, 
MA). Figs. 2 and 3 show sets of simultaneously recorded 
data from each subject. The gains for the different channels 
were set individually prior to data collection. 
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Fig. 2. Continuous signals acquired simultaneously from (a) a macaque and (b) an owl monkey demonstrate the functionality of our system in actual 


biological systems. 


4. Discussion 
4.1. System considerations 


Our telemetry system is highly versatile, and may be con- 
figured in both software and hardware. Software changes in 
the CPLD could alter the sampling rate, the channel selec- 
tion procedures, and the ADC resolution. Different batteries 
may be used depending on size/weight and lifespan require- 
ments. 


4.2. Digital design 


Digitizing the data provides two major benefits. Firstly, 
digitized data is well suited to the type of on-board sig- 
nal processing which will likely be incorporated in the fu- 
ture. Algorithms for procedures, such as spike detection and 
spike sorting will be easier to implement and more versatile 
in the digital domain. The second benefit of the digital ap- 
proach is in the telemetry. Digital telemetry provides immu- 
nity to transmission channel noise, and will ultimately re- 
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Fig. 3. Sorted action potentials derived from simultaneously acquired data from (a) a macaque and (b) an owl monkey. Signals were sorted after data 


collection using offline spike sorter (Plexon, Inc.). 


sult in lower bit-error rates. Furthermore, the digital teleme- 
try hardware may easily be be replaced with a standalone 
wireless module (such as Bluetooth® (Morrow, 2000) or 
Ultrawideband (Barrett, 2001)) depending on system re- 
quirements, such as battery life and transmission range. 


4.3. Latency 


This system was developed as part of a real-time neu- 
ral prosthesis, where data latency can cause delays in com- 


mand execution (Donoghue, 2002). The latency of our sys- 
tem is 680 1s, measured from the analog input through the 
FIFO input. The latency after the FIFO is software depen- 
dent; for our configuration, the delay from FIFO input to 
output ranged from 100 wu to 50ms depending on the num- 
ber of active channels, the ADC resolution, and the lo- 
cation of a particular sample within its respective packet. 
The maximum delay through the FIFO can be reduced in 
software by reducing the number of bytes stored in the 
FIFO before transmission. However, the effects of such a 
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reduction on the system throughput have not been mea- 
sured. 


4.4. Bandwidth 


Our system transmits neural signals in their entirety, with- 
out isolating single units or action potentials. This allows for 
faithful transmission of subthreshold depolarizations and hy- 
perpolarizations that may be important indicators of synapse 
behavior (Covey, 2000). However, if single units are detected 
and spike sorted before transmission, the data rate could be 
decreased by at least one order of magnitude (depending on 
signal quality and detector efficiency), thus increasing the al- 
lowable number of channels over to over 100 (Moxon et al., 
2001). The number of channels could be increased further 
by using multiple wearable telemetry units on the same sub- 
ject; the IEEE 802.11b standard allows for multiple devices 
to operate in close proximity. 


4.5. Multipath 


Multipath echoes can adversely affect the signal quality 
of any wireless transmission network (Rappaport, 1996). In 
our system, excessive multipath may cause data packets to 
be missed during transmission. Since UDP network sockets 
do not retransmit missed packets, any such data will be per- 
manently lost. Our device was tested in vivo in a standard 
animal laboratory setting that was not optimized to reduce 
multipath reflection surfaces, and was able to transmit 12 
channels at a range of 9m without any lost data. Multipath 
effects could possibly be mitigated by transmitting the data 
via a TCP socket that would retransmit lost packets. How- 
ever, this would increase overhead and would reduce the 
maximum number of transmission channels. 


5. Conclusion 


A multichannel neural telemetry system capable of con- 
tinuously transmitting 12 of 16 channels has been demon- 
strated. The system architecture is flexible and may be 
modified to accommodate new radio technologies or en- 
hanced neural signal processing techniques. 
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Optimizing a Linear Algorithm for Real-Time Robotic 
Control using Chronic Cortical Ensemble 
Recordings in Monkeys 


Johan Wessberg and Miguel A. L. Nicolelis 


Abstract 


Mi Previous work in our laboratory has demonstrated that a 
simple linear model can be used to translate cortical neuronal 
activity into real-time motor control commands that allow a 
robot arm to mimic the intended hand movements of trained 
primates. Here, we describe the results of a comprehensive 
analysis of the contribution of single cortical neurons to this 
linear model. Key to the operation of this model was the 
observation that a large percentage of cortical neurons located 
in both frontal and parietal cortical areas are tuned for hand 
position. In most neurons, hand position tuning was time- 
dependent, varying continuously during a 1-sec period before 
hand movement onset. The relevance of this physiological 
finding was demonstrated by showing that maximum contri- 
bution of individual neurons to the linear model was only 
achieved when optimal parameters for the impulse response 


INTRODUCTION 


In the past, several studies have provided data that 
support the hypothesis that cortical activity could one 
day provide the necessary input signals required for 
controlling a neuroprosthetic device (Schmidt, 1980; 
Fetz & Cheney, 1978; Fetz & Baker, 1973; Fetz & 
Finocchio, 1972, 1975; Fetz & Finnocchio, 1971). This hy- 
pothesis assumes that voluntary cortical activity, related 
to intended arm movements, could be sampled from 
healthy parts of the motor cortex and, through the 
employment of real-time models, be translated into 
intended movements either by directly controlling arti- 
ficial devices, such as a robot arm (Chapin, Moxon, 
Markowitz, & Nicolelis, 1999), or by inducing coordinat- 
ed muscle contraction in the patient’s paralyzed limbs 
(Kilgore, Peckham, & Keith, 2001, chap. 2). Studies in 
rodents (Chapin et al., 1999) and primates (Serruya, 
Hatsopoulos, Paninski, Fellows, & Donoghue, 2002; 
Taylor, Tillery, & Schwartz, 2002; Wessberg, et al., 2000) 
have demonstrated that there is merit in this propo- 
sition and that further experimental work should be 
able to clarify whether such an approach may become 
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functions describing time-varying neuronal position tuning 
were selected. Optimal parameters included impulse response 
functions with 1.0- to 1.4-sec time length and 50- to 100-msec 
bins. Although reliable generalization and long-term predic- 
tions (60-90 min) could be achieved after 10-min training 
sessions, we noticed that the model performance degraded 
over long periods. Part of this degradation was accounted by 
the observation that neuronal position tuning varied signifi- 
cantly throughout the duration (60-90 min) of a recording 
session. Altogether, these results indicate that the experimen- 
tal paradigm described here may be useful not only to 
investigate aspects of neural population coding, but it may 
also provide a test bed for the development of clinically useful 
cortical prosthetic devices aimed at restoring motor functions 
in severely paralyzed patients. Hi 


clinically relevant in the near future. Perhaps more 
conclusively, some of these recent animal studies have 
indicated that, at the very least, the ability to implement 
operational real-time brain—machine interfaces (BMIs), 
which define the core of the envisioned neuropros- 
thetics, has created a very powerful new experimental 
paradigm in systems neuroscience (Nicolelis & Ribeiro, 
2002; Nicolelis, 2003). Because in this type of study the 
experimenter can control both the real-time models 
used to analyze patterns of neuronal activity and the 
type of “artificial” sensory feedback returned to sub- 
jects, such BMIs offer a unique opportunity to investi- 
gate causal dependencies between neuronal activity and 
animal motor behavior (Nicolelis & Ribeiro, 2002). 

The development of techniques for chronic, multisite 
recordings of ensembles of single cortical (and/or sub- 
cortical) neurons using multielectrode arrays has be- 
come a critical component of attempts to test the 
experimental and clinical potential of BMIs. For exam- 
ple, using microwire arrays chronically implanted in the 
motor cortex and thalamus, Chapin et al. (1999) 
achieved one-dimensional cortical control of a robot 
arm in rats. In another study, we have demonstrated 
how a similar technique can be used to replicate, in a 
robot arm, more complex 1-D-3-D hand movements 
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generated by trained New World monkeys (Wessberg 
et al., 2000). In this study, we implanted multiple 
chronic microelectrode arrays in motor, premotor, and 
parietal cortical areas in monkeys involved in the gen- 
eration of the commands for reaching arm movements 
(Batista, Buneo, Snyder, & Andersen, 1999; Georgopou- 
los, Kalaska, Caminiti, & Massey, 1982; Weinrich & Wise, 
1982; Fetz & Cheney, 1978; Mountcastle, Lynch, Geor- 
gopoulos, Sakata, & Acuna, 1975; Evarts, 1966). Single- 
unit activity from ensembles of cortical neurons was 
recorded as the monkeys performed 1-D-—3-D arm move- 
ments. By using either linear or nonlinear mathematical 
techniques, we were able to transform the recorded 
ensemble activity in real time to a control signal that in 
turn was used to generate movements in a robot arm. 
We demonstrated that the produced robot arm move- 
ments closely mimicked the arm movements made by 
the monkeys. Thus, in essence these experiments dem- 
onstrated the feasibility of the idea that chronically 
implanted microwire arrays and real-time mathematical 
transformation of neural ensemble activity may one day 
allow severely disabled human patients to control the 
movements of a prosthetic limb or other devices. Since 
then, other reports have reproduced our findings and 
demonstrate the feasibility of closed-loop control of a 
neuroprosthetic device in macaque monkeys (Serruya 
et al., 2002; Taylor et al., 2002). 

In our previous study we explored a large variety of 
linear and nonlinear techniques. We observed, however, 
that nonlinear methods based on artificial neural net- 
works (ANNs) did not significantly outperform a simpler 
method based on multiple linear regression. This finding 
motivated us to analyze the linear regression method for 
real-time transformation of cortical signals in greater 
detail. Because there are several parameters that have 
to be chosen by the experimenter in this particular 
application, several questions remained open regarding 
the selection of optimal values of parameters for this 
linear model. For example, the temporal resolution and 
length of the data window used for estimating position 


at each point in time has to be chosen, as well as the 
amount of data required to fit the model. In the present 
study, we analyzed the optimal choice for such parame- 
ters. We also evaluated the performance of the linear 
model for predicting arm position at a future point in time 
with time shifts in the order of hundreds of milliseconds. 

In contrast to an ANN model, where the ‘“‘black-box”’ 
nature of the algorithms make it difficult to directly 
analyze the exact contributions of the individual neu- 
rons, the linear model allowed a detailed analysis of how 
different neurons contributed to the derived control 
signals. Some of these findings have been presented 
previously in abstract format (Nicolelis, Wessberg, Kralik, 
& Beck, 2001; Wessberg, Crist, & Nicolelis, 2001). 


RESULTS 


Two monkeys were trained to make left-right move- 
ments of a joystick. Ten recording sessions were ana- 
lyzed in each of the two animals. The duration of the 
sessions were 45-120 min, and 34-107 neurons were 
recorded per session. Figure 1 shows examples of the 
recorded data for the two animals while they made 
movements of a joystick along a left-right path. Figure 1A 
shows raster plots of the discharges of 98 neurons 
recorded along with movements in Monkey 1. Figure 1B 
shows 41 neurons recorded in Monkey 2. In both 
monkeys the movements were irregular, with varying 
movement sizes and speeds, varying starting points for 
the movements, and varying delays of position holding 
between the movements. The movements of Monkey 2 
tended to occur at slightly lower rates, but were other- 
wise similar to those of Monkey 1. The large variability of 
the animals’ movements makes it possible to test whether 
algorithms for predicting position from simultaneous- 
ly recorded neuronal signals can generalize beyond a 
limited set of trajectories. Hence, in the present study 
we considered the large variability of the movements 
greatly advantageous compared to a set of stereotypical 
movements recurring at fixed intervals. 


Figure 1. Examples of 
simultaneously recorded 
left-right hand position and 
neuronal data in two owl 
monkeys. (A) 94 cells recorded 
in Monkey 1, (B) 41 cells 
recorded in Monkey 2. 
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Figure 2. Impulse response 
functions in fitted linear A 
models, (A) Monkey 1, (B) 
Monkey 2. Error bars show 
95% confidence limits for the 
weights obtained by bootstrap 
estimation. A significant 
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variations in the magnitude of 
the impulse response. 


Coding of Arm Position by Cortical Cells 


In no case did inspection by eye of raster plots such as 
those shown in Figure 1 reveal prominent signs of 
modulation of cell activity in relation to the movements. 
As a first step in analyzing the extent cortical cells coded 
for the position of the hand along the left-right move- 
ment path we calculated simple linear regression be- 
tween the activity of single units, counting the number 
of discharges in 100-msec bins, and hand position. Linear 
correlation between unit activity and hand position was 
significant in 39.8% of the recorded neurons in our two 
monkeys (p < .01, mean, range for individual sessions 
26.8-54.3%). Both positive and negative correlations 
were found, and significantly correlated units were found 
in all cortical areas. This shows that some cortical neu- 
rons modulate their firing rate according to hand posi- 
tion. However, this simple analysis can only reveal 
correlation that is near synchronous with the current arm 
position. In addition, cortical activity up to some limit 
backward in time should be taken into account, for 
example, preparatory cortical activity can precede move- 
ment by hundreds of milliseconds (Wise, Boussaoud, 
Johnson, & Caminiti, 1997; Alexander & Crutcher, 1990; 
Kurata & Wise, 1988). This can be achieved using a linear 
regression model where an impulse response function, or 
in other words, a set of weights, is calculated for each 
cortical unit. The activity of each unit is then convolved 
with its impulse response, and the resulting data vectors 
summed, to provide predictions of arm position. Hence, 
the weights in the impulse response functions are meas- 
ures of the time-varying correlation between unit activity 
and arm position. 

Figure 2 shows 10-bin impulse response functions 
for 10 cortical units. The linear model was derived using 
10 min of data at a temporal resolution of 100 msec (i.e., 
10 Hz). At this resolution, 10 bins of data correspond to 
1 sec of time. A few units (overall 8.5%) showed a 
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relatively simple relationship with position, with an initial 
positive or negative weight with successively decreasing 
amplitude. This roughly corresponds to a positive or 
negative correlation as revealed by the correlation anal- 
ysis, above. The majority of cortical units in both animals 
and all sessions showed more complex impulse re- 
sponse functions. Units typically showed a time-varying 
correlation with position within the impulse response, 
with many units only having significant weights for 
part of the 10-point function, or a reversal between po- 
sitive and negative weights. In 10 experimental sessions 
in each animal, 56% of the cells (pb < .05, mean, range 
39.0-72.0%) had at least one statistically significant 
weight in the impulse response function. Cells with pre- 
dominantly positive weights (mean 22%, range 12.6— 
33.3%) were as common as cells with predominantly 
negative weights (mean 22%, range 14.1-30.8%). 12% of 
the cells (mean, range 2.1-27.0%) showed a reversal 
between positive and negative weights with at least one 
significant weight in each direction in its impulse re- 
sponse. The patterns of impulse responses hence showed 
a marked heterogeneity within the ensembles of cortical 
cells. This was true for all cortical areas recorded in two 
monkeys. There was no tendency for units from one 
particular cortical area to show one pattern of impulse 
response functions, although the sample for each corti- 
cal area in the present study is too small to draw any 
such comparative conclusions. 

It is not trivial to interpret the wide range of patterns 
of the impulse response functions. One possibility is 
that the variable chosen to fit the linear models, for ex- 
ample, position, does not correlate particularly well with 
a cell’s firing, but that a cell is rather correlated with, for 
example, hand velocity or acceleration. It is possible to 
fit linear models using either velocity (first derivative of 
position) or acceleration (the second derivative) and 
applying exactly the same algorithm as for position. 
Impulse responses obtained for a set of cells when fitting 
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position, velocity, or acceleration are shown in Figure 3. 
In some cases, a cell with a complex impulse response 
for position (e.g., Unit 3 in Figure 3) could show strong 
velocity dependence at lower numbered weights in the 
impulse response. However, the reverse was also true, 
for example, Unit 1, which has a strong position corre- 
lation at lower numbered weights, but with a complex 
time-dependent impulse response function for velocity. 
Importantly, in no cases we observed that a complex 
impulse response in the time domain was ‘‘explained”’ 
by a cell having a simpler correlation with velocity or 
acceleration. Instead, this analysis revealed that virtually 
all significantly modulated cells were correlated with 
each of these three kinematic variables in a complex, 
time-dependent fashion. 

Because a multiple regression algorithm was used to 
derive the impulse responses, it may be argued that the 
individual weights are not independent, but were affect- 
ed by all the variables included in the model, that is, all 
recorded cells. To test whether including a variable 
number of cells affected the shapes of the impulse 
response, models were fitted for the same data sets 
using, first, all available cells, then, only 10 cells that 
were strongly modulated by hand position, and finally 
only a single cell, one at a time. As it has previously been 
described, fitting models with subsets of all cells always 
result in less accurate overall fit, as well as less accurate 
predictions when using the models to predict position 
with new data (Wessberg et al., 2000). In contrast, the 
shapes of impulse responses for individual cells were 


only minimally affected. Figure 4 shows the results for 
three examples of cells from Monkey 1. In general, the 
magnitude of the weights tended to vary inversely with 
the number of cells included in the model. (Increased 
absolute weights can be expected when fewer variables 
are available to explain a fixed amount of variance in the 
dependent variable). However, the direction of the 
weights as well as the general shape of the impulse 
responses remained constant, also for units with com- 
plex impulse responses. Hence, these impulse responses 
probably reflect the underlying physiological relation- 
ship between the cell’s pattern of activity and the chosen 
movement parameter. 


Evaluation of the Linear Regression Model 


As previously reported, our linear models consistently 
predict arm position for new data recorded in the same 
experimental session. Figure 6A shows the performance 
of linear models derived during 10 sessions in each 
monkey as a function of the number of recorded units. 
The R’ scores are the squared linear correlation between 
predicted and actual linear arm position for 10-min 
segments of data that were not used to fit the models. 
Hence, these R* scores measure the average perfor- 
mance of the models for predicting position when using 
new data. It is important to note that there are no 
discrete movement trials in the records, but that the 
position of the arm is being predicted continuously. 
There was considerable day-to-day variation between 
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Figure 3. Impulse responses for models fitted to corresponding records of hand position (top), velocity (middle), and acceleration (bottom). Each 
column shows impulse responses of single cells for models fitted to each of the three kinematic parameters. 
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Figure 4. Analysis of the 
dependence of impulse 
response functions on other 
cells included in the model. 
Impulse responses are depicted 
for the same three units for: (A) 
linear models fitted using all 
available cells in a session; (B) 
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sessions, but there is also a significant linear correlation 
between recorded number of units and overall perfor- 
mance of the model (r = .65; p < .05). 


Varying the Parameters of the Linear Model 


Figure 5B and C shows the effect of varying length of the 
impulse response functions, or equivalently the width of 
the “data window” used for predicting each point in the 
position record (i.e., varying 7 in the linear equation; see 
Methods). A temporal resolution (bin width) of 100 msec 
was used, and 10-min data segments were used to fit and 
evaluate the models. Figure 5B shows three sample 
sessions in each animal, and Figure 5C shows averages 
for 10 sessions in each animal. To enable averaging in 
Figure 5c, data for each session have been normalized so 
that the R* score for a 10-bin window for each session 
equals 100%. Note that using a window of single data bin 
yields models that provide predictions with R* scores of 
on the average only 40% compared to a 10-bin window. 
With a single bin, only activity that is almost synchro- 
nous with current arm position is used in the model, 
analogous to basing a model on the simple linear 
correlation between cell activity and position. The per- 
formance of the model peaked at 10- to 14-bin data 
windows, corresponding to data windows of 1.0 to 
1.4 sec for 100-msec bin width. For the further analyses, 
a 1.0-sec window was used if nothing else is noted. 
Next, we analyzed the effect of varying the length of 
the segment of recorded data used to derive the linear 
models. Figure 5D and E shows predictions when the 
length of the data segment was varied from 1 to 20 min. 
Figure 5D shows three sample sessions in each animal, 
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and Figure 5E shows averages for 10 sessions in both 
animals. In general, when very short data segments of 
1 to 2 min were used the predictions obtained for new 
data were unreliable, and low R? scores were obtained. 
Increasing quality of the predictions was obtained by 
increasing the data segment from 2 to 10 min. There was 
little improvement in the predictions from models when 
data segments longer than 8-10 min were used, and 
10-min segments were normally used in all analyses in 
the present study. 


Varying the Temporal Resolution 


The 10-Hz resolution achieved with 100-msec bin widths 
may be considered very low for accurately representing 
the movements of a limb in general. Power spectra of 
the position records showed that the variance of the 
position records above 4 Hz was negligible. Hence, 
10-Hz was sufficient to represent most of the variance 
of the movements in the present study. Nevertheless, it 
is important to evaluate the linear model for higher 
temporal resolution. The cost of increasing temporal 
resolution is a much higher computing load, with the 
particular limiting factor usually being the amount of 
computer primary memory available for storing the large 
matrices used when fitting the linear regression models. 
Overall, differences between predictions obtained 
with 50- and 100-msec bin widths were negligible and 
nonsignificant. The mean difference between 50- and 
100-msec resolution in terms of R* scores was —.0008 
(range —.012 to .032). Analysis of the errors in the 
predicted position trajectories showed no differences. 
In addition, we compared using 25-, 50-, and 100-msec 
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Figure 5. R? for predictions as a function of varying the parameters of the linear model. (A) Varying number of recorded cells in 10 consecutive 
sessions in both animals. (B) Varying the number of bins used in the model: absolute R? scores for three sample sessions in each animal. 

(C) Average normalized R* scores for 10 sessions in each animal. 100% corresponds to the score for 10-bin models. Ten minutes of data were used 
to fit the linear models. (D) Varying the length of the data segment used to fit the model: absolute R* for three sample sessions. (E) Average 
normalized R? scores for 10 sessions in each animal. 100% corresponds to the score for using 10 min of recorded data to fit the model. Ten 100-msec 


bins of data were used in the models. 


bin widths (i.e., 40, 20, and 10 Hz, respectively). To fit a 
40-Hz model using the available computing power, a 
model data window of only 0.6 sec was used instead of 
the standard 1.0-sec window, and in addition only 5 min 
of data was used to fit the model rather than the usual 
10. Hence, the models yielded only modest predictions 
in the range of R? = .3. In any case, performance of the 
10-, 20-, and 40-Hz models was nearly identical, with the 
largest observed difference in terms or R* between 40 
and 10 Hz being —.014. This is not surprising given that 
movements were accurately represented already using 
10-Hz signals in the present study, but it provides an 
indication that given sufficient computing power, a 
linear regression model should be able to handle move- 
ments with other characteristics, with significant vari- 
ance up to 15—20 Hz. 


Predicting Position into Future Time 


It is possible to modify the linear model so that the 
recorded neural activity can predict position “into the 
future,” that is, for a position that will occur with a small 
fixed time shift forward from the last recorded neuronal 
discharges. Such a modification of the linear model may 
be important for an application where the derived 
position predictions are used to control a robot arm 
or other device, and where continuous sensory feedback 


about the motion of the robot will permit an animal or 
patient to learn to functionally control the device. The 
presence of a time delay in the robot system would 
make learning to control the device significantly more 
difficult, and the prediction algorithms should ideally 
compensate for any such fixed time delays. Figure 6A 
and B shows sample records from both animals of 
predictions made with a constant time shift of 100 msec 
for predicting position into future time. Figure 6C shows 
the average performance of the linear model for 10 ses- 
sions in both monkeys when the time shift was varied. 
The R* scores have been normalized so that R* for 
synchronous predictions (i.e., no time shift; the stan- 
dard linear model) equals unity. Predictions with a time 
shift of 100 msec had an accuracy of 92% compared to 
synchronous predictions (range 85-97%). At 200 msec, 
84% accuracy was achieved (range 70-93%). This shows 
that obtaining technically useful predictions into future 
time are feasible using the linear model, at least for shifts 
up to 100 msec or less. 


Adaptive versus Fixed Linear Models 


We have demonstrated that it is possible to obtain 
accurate predictions of arm position all through record- 
ing sessions lasting 90 min or more. In a previous study, 
an adaptive algorithm was used, and predictions for all 
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points in time during the sessions were based on models 
that had been fitted to recently recorded 10-min seg- 
ments of data (Wessberg et al., 2000). The alternative is 
to use a nonadaptive model, where the impulse re- 
sponse function weights are fixed. Figure 7A shows the 
performance of such a nonadapting algorithm for two 
sample sessions. A model was fitted at the beginning of 
the sessions, and the performance of the model for 
predicting arm position was evaluated over time. There 
was a steady degradation in the quality of the predic- 
tions over the experimental session. To the right in the 
panel of Figure 7A there is also data for a second model, 
fitted during the 30- to 40-min segment. These newly 
fitted models initially yielded predictions at the same 
level as the first models, but again predictions degraded 
progressively over time. Figure 7B shows the average 
results for 10 consecutive sessions in both animals. To 
enable averaging and comparisons, all R? scores have 
been normalized so that the score for predictions during 
the first 10-min segment after the model was fitted 
equals 100%. The graphs show that models fitted at all 
points in the sessions will degrade slowly over time. On 
the average, a model fitted at the beginning of a session 
could be expected to stay within 90% of the initial score 
for 50 min or more, and to drop to 50% of the initial 
performance within 70-80 min. Figure 7B also reveals a 
tendency for models fitted later in the session to dete- 
riorate slightly faster than models fitted more early in 
the session. 

One possibility is that the demonstrated degradation 
of the predictions reflects changes in the animals’ be- 
havior over the duration of a session. However, inspec- 
tion of the records of animals’ movements and the 


corresponding predictions of arm position revealed no 
overt differences. An example is shown in Figure 7C and 
D (5 min into the session in C; 95 min in D). This was 
further analyzed by comparing power spectra of the arm 
position records for longer segments of data. Figure 7E 
shows the power spectra for position for the 0-10 and 
90-100 min data segments in the same session and in 
Figure 7C and D. The very close similarity of the spectra 
shows that the extent, speed, and frequency of the 
animals’ movements were nearly constant from the 
beginning to close to the end of the session. 

If behavior is constant, an alternative explanation for 
slow deterioration in the fitted models’ capacity to 
provide predictions of arm position would be that the 
relationship between the activity of the recorded units 
and arm position is variable. In other words, the coding 
of arm position of single units should show slow 
changes over time. Figure 8 shows examples of impulse 
response functions for four cortical units at four differ- 
ent points during one experimental session. As shown 
in the examples, changes were observed in the magni- 
tude in one or more weights (Unit 19), or in the overall 
shapes of the impulse responses (Units 25, 55). Finally, 
in a few units, we observed reversals from positive to 
negative correlation in some weights (Unit 76). As in 
the examples in Figure 8, changes were not limited to 
single weights but affected several contiguous weights, 
or the whole impulse response. Moreover, the ob- 
served changes did not appear to be random fluctua- 
tion from one 10-min segment to the next but they 
seem to be gradual over time. Hence, even though the 
animals’ motor behavior was constant, many units 
steadily and slowly changed their coding of arm posi- 
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Figure 6. Predictions obtained with a fixed time offset into future time. (A) and (B) show examples of observed and predicted position in Monkey 
1 and 2, respectively, using a time shift of 100 msec. (C) A comparison of average normalized R* scores for 10 consecutive sessions in each animal 
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Figure 7. Evaluation of predictions from linear models as a function of session time. (A) Absolute R? scores for predictions in two experimental 
sessions. Models were fitted with data from 0 to 10 min into the session (left set of lines) and 30-40 min (right set of lines). (B) Normalized 
average R’ scores for 20 sessions in two animals. 100% corresponds to predictions obtained for the first 10 min after the models were fitted. Models 
were fitted using different data segments as indicated. (C) and (D) Samples of actual movements and predictions at 5 and 95 min into an 
experimental session. (E) Corresponding power spectra for position data from 0-10 min (black line) and 90-100 min (blue line). 


tion over the extent of the recording session. This 
would provide an explanation for the observed degra- 
dation of the predictions provided by the linear models. 
We noted that the gradual changes were not a charac- 
teristic of the impulse response function analysis, but 
were also observed in simpler measures of the relation- 
ship between units’ activity and arm position such as 
simple linear correlation between cell activity and arm 
position. 

Slow changes in the impulse response functions were 
observed in cells in all recorded cortical areas. Using 
bootstrap statistical methods, we analyzed the propor- 
tion of units showing significant weight variations in 
10 sessions in both animals. We found that 51-92% 
(mean, 78%) of the cortical units showed a significant 
change over the experimental session in at least one 
weight at the p < .01 significance level. The slightly 
lower percentages were observed in the shortest ses- 
sions that lasted only 45-60 min. Hence, we tentatively 
suggest that such changes were commonplace given a 
long enough observation time. In summary, these re- 
sults show that the contribution of single units to the 
coding of arm position, obtainable at the ensemble level, 
changes slowly over time with a timescale in the order of 
tens of minutes. 


DISCUSSION 


The Linear Model for Neuroprosthetic Control 


In this and a previous study, we have shown how 
accurate predictions of hand position in space can be 
obtained in real time using a simple linear model based 
on multiple linear regression of the spike trains re- 
corded from ensembles of cortical neurons (Wessberg 
et al., 2000). We previously demonstrated how the 
predictions could be used as a control signal to con- 
trol the movements of a robotic arm, thus demon- 
strating the feasibility of constructing neuroprosthetic 
interfaces using this mathematical algorithm. We have 
here explored the optimal parameters for such an 
algorithm when used for predicting relatively simple 
arm movements in monkeys: Our results suggest that 
around 1 sec of neuronal data should be used to 
construct each real-time estimate of position, and 
around 10 min of data may be needed to estimate the 
model parameters. The importance of this finding is that 
the algorithm is efficient enough to be implemented in 
current PC computer-type technology, also for concur- 
rent recordings of several hundreds of cortical cells. 
Hence, the algorithm should provide a useful starting 
point for portable or even implantable devices. 
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We have further shown that 20-Hz resolution (50-msec 
bins) can be achieved directly using an unmodified 
algorithm, and that 40-Hz resolutions should be possible 
to implement in faster microprocessors. This should be 
sufficient for controlling a neuroprosthetic device to 
assist or replicate most natural arm movements. We have 
also demonstrated that the linear model can, with only 
small modification, be used to predict position ahead 
of the present position of the limb. Reasonable quality 
of the predictions can be achieved up to 100-200 msec 
into ‘future time.’ Such predictions may be important 
to compensate for any fixed time delays in an applica- 
tion where a robot arm or other device is used under 
ongoing visual or other sensory feedback by the experi- 
mental animal or patient. This may be a necessary to 
enable an animal or patient to learn to use an advanced 
neuroprosthetic device. 


Alternative Approaches 


Several alternative algorithms for converting neuronal 
activity into a control signal for neuroprosthetic appli- 
cations have been proposed (Schwartz, Taylor, & Helm 
Tillery, 2001). Some of these rely on identifying given 
physiological characteristics of the recorded cells, such 
as cells broadly tuned to direction of movement in space 
(Salinas & Abbott, 1994; Schwartz, 1994; Georgopoulos, 
Kettner, & Schwartz, 1988), and cells coding speed of 
movement (Moran & Schwartz, 1999). In all these 
approaches, it is necessary that the identified properties 
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of the neurons generalize from the set of trials used to 
identify the firing properties to the actual repertoire of 
movements used during robotic control. The linear 
algorithm outlined in the present study relies on the 
use of continuous records of movement and neuronal 
data rather than discrete set of movement trials with 
prespecified characteristics such as movements to a 
prescribed set of targets, or movements made at a set 
of prescribed speeds. Thus, the animal’s natural move- 
ments produced during meaningful motor tasks can be 
used both for estimation of the model parameters and 
evaluation of the performance of the model. The tech- 
nique also uses a totally random and unselected sample 
of cells. This means that all the available information in 
the recorded sample (so far as it can be extracted using a 
linear algorithm) is used, rather than relying on a subset 
of cells with identified physiological properties. 

Other suggested alternatives to the linear model rely 
on different types of nonlinear techniques, including 
maximum likelihood methods based on probability the- 
ory (Brown, Frank, Tang, Quirk, & Wilson, 1998), pat- 
tern recognition (Isaacs, Weber, & Schwartz, 2000), or 
different ANN techniques. In the previous study, we 
examined a large number of feed-forward ANN architec- 
tures and discovered that, at least for simple arm move- 
ments, the best ANNs would on the average only 
perform marginally better than the linear regression 
model. It is also important to note that the linear 
regression method is efficient enough to be used adap- 
tively: Model parameters can be estimated during an 
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ongoing experimental session, and the parameters can 
be adaptively modified during the session if necessary to 
maintain the quality of the predictions. When ANN 
techniques were used in real-time during ongoing ex- 
perimental sessions, the minor advantages in quality of 
predictions were usually offset by the stringent time 
requirements of an algorithm where model parameters 
are to be estimated and adapted during an ongoing 
experimental session. 

In summary, we have shown that the suggested linear 
method can provide estimates at high resolution (40 Hz 
using personal computers), can be extended to multiple- 
dimensional control (Wessberg et al., 2000), and is 
currently only marginally inferior to nonlinear algo- 
rithms based on neural networks. It can simply be 
modified to predict “into the future” for further devel- 
opments involving visual or tactile feedback (Nicolelis, 
2003). Furthermore, the algorithm has been proven to 
work in actual implementations of on-line robotic con- 
trol. A similar linear method was used in a study where 
monkeys controlled a computer screen cursor in real 
time using cortical signals (Serruya et al., 2002). 


Coding of Movement-Related Parameters 
in Cortical Cells 


The fact that it is possible to derive real-time estimates of 
arm position from a random sample of cortical cells in 
motor, premotor, and parietal cortex shows that motor 
control signals for arm movements in space are likely 
widely distributed and appear nearly simultaneously in 
multiple sensory-motor cortical areas (Georgopoulos 
et al., 1982; Weinrich & Wise, 1982). The linear regres- 
sion algorithm used in the present study has the advan- 
tage over most nonlinear techniques that the details of 
the contribution of individual cells can be directly 
assessed. Two associated findings in the present study 
have bearing on the issue of coding of movement 
parameters by cortical cells: First, derived predictions 
of position improve by, on the average, a factor of 2.5 
when 1 sec of activity is used compared to when only 
near-synchronous cortical activity is considered. Second, 
the impulse response functions for all neurons were not 
interpretable in terms of a simple tuning function 
(corresponding to negative or positive correlation for a 
one-dimensional movement). While a few cells exhibited 
positive or negative correlation with arm position using 
the simplified histogram analysis, the optimal correla- 
tion between the activity of single randomly sampled 
cortical cells and arm position in space was shown to 
have a complex time course for most cells. The first 
finding highlights the importance of the latter: The 
complex impulse responses are important because the 
improvement in predictions is achieved when they are 
taken into account in the model. Because the shapes of 
impulse responses remained constant when subsamples 
of cortical neurons were used to fit the models, the 


complex impulse responses did not seem to reflect any 
artifact of the multiple linear regression technique, but 
instead they may depict true correlations between neu- 
ronal activity and arm position. Cells with significant 
impulse response weights at time lags up to 1 sec were 
commonplace in all recorded cortical areas. Impulse 
responses were equally complex when velocity or accel- 
eration was used instead of position. 

One implication of these results is that analyses of 
neuronal activity at near-zero time lags only provide a 
very limited view on the actual relationships between 
cortical activity and a parameter of movement. Although 
movements in the present study were limited to a single 
dimension, it can be argued that the present findings 
have implications for the interpretation of previous 
work where cell firing within a limited time window 
has been examined, or when time-varying activity has 
been examined at coarser timescales than 100-msec bins. 

It may be argued that the complex impulse response 
functions may in part be due to the complex movement 
patterns, and that analyses were not limited to sets of 
trials with more well-defined or restricted arm move- 
ments. On the other hand, the present findings could be 
taken to indicate that the true correlation between 
cortical activity and a movement parameter such as 
arm position is more complex for more unrestricted 
movements than hitherto commonly believed. Further- 
more, the present results show what can be expected 
when a totally random sample of cells is examined, 
without preselecting cells with particular, identified 
response properties in relation to the movements. 

Previously, we have shown that different parameters 
of movement could be estimated using the same neural 
population. It was found that predictions of hand posi- 
tion in space were slightly better than predictions of 
hand velocity or acceleration. Intrinsic parameters such 
as muscle activity or joint angles instead of the position 
of the hand in space were not made in this study. 
However, it is likely that joint rotations correlate strong- 
ly with hand position in the present task: The left-right 
hand movements were largely achieved by horizontal 
shoulder rotation, coupled with small extension—flexion 
movements of the elbow. In addition, preliminary evi- 
dence collected in our laboratory suggests that the same 
linear model could be successfully used to predict EMG 
activity (Santucci, Kralik, & Nicolelis, 2002). 

The literature contains arguments as to what param- 
eters are represented in motor cortical cells (direction, 
speed, position), although it has been argued that no 
single parameter is likely to be represented in a ran- 
domly selected population of cells (Todorov, 2000), or 
conversely, that information about several concurrent 
movement parameters are superimposed. It has been 
pointed out that cortical neurons involved in processing 
motor commands are modulated by many different 
aspects of movement and are likely modulated by non- 
motor factors as well Johnson, Mason, & Ebner, 2001). 
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Hence, recorded neuronal activity likely reflects the 
working of multiple sensory and motor neuronal net- 
works that are continuously and concurrently active 
during the planning and execution of arm movements. 
The identification and analysis of such subnetworks 
of cortical cells during natural motor behaviors is an 
important challenge for future studies in which the 
activity of increasing numbers of cortical cells will be 
chronically recorded in awake behaving animals. The 
technique of fitting a movement parameter using the 
multiple linear regression technique and examination of 
the time course of impulse responses may be an impor- 
tant tool in the investigation of subnetworks among the 
recorded cells. For example, it may be possible to dis- 
tinguish between cells that are more prominently corre- 
lated to a parameter of movement at shorter or longer 
time lags, tentatively reflecting the cells’ involvement at 
earlier or later stages in the processing of a motor 
command. 


Variation of Movement Coding over Time 


We showed that the predictions we obtained from linear 
models with fixed weights slowly degraded over time. 
One possible reason would be that the behavior of the 
animals was changing as they were learning the task or 
adapting to the experimental situation, and that the 
model was unable to correctly predict movements out- 
side a limited movement repertoire. However, we could 
not observe any changes in the details of the animals’ 
movements during the sessions. The alternative inter- 
pretation is that there were gradual changes in the 
coding of arm position in a proportion of the recorded 
cortical neurons. This was confirmed when we examined 
the impulse response functions for single cortical neu- 
rons as a function of experimental session time. Such 
changes appeared to be slow and gradual, occurring 
successively over tens of minutes, and apparently wide- 
spread within the neuronal population, affecting most of 
the units provided the observation time was sufficient. 
It is well established that the functional properties of 
motor cortical neurons in animals and man can change 
rapidly, for example, in response to peripheral nerve 
injury or central lesions, or repetitive stimulation (Sanes 
& Donoghue, 2000). Functional plasticity of motor 
cortex has been observed in monkeys during visuomo- 
tor learning or adaptation (Wise, Moody, Blomstrom, & 
Mitz, 1998; Mitz, Godschalk, & Wise, 1991) or during 
acquisition of new motor actions (Laubach, Wessberg, 
& Nicolelis, 2000; Nudo, Milliken, Jenkins, & Merzenich, 
1996). Furthermore, it has been shown that single 
motor cortical cells’ coding of movement direction 
can change within the experimental session when a 
monkey is trained to adapt to an external load applied 
during arm movements (Li, Padoa-Schioppa, & Bizzi, 
2001; Gandolfo, Li, Benda, Padoa Schioppa, & Bizzi, 
2000), or over days when monkeys are trained to 
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control a visual target with the activity of cortical 
neurons fed through a fitted model (Taylor et al., 
2002). Moreover, preliminary results indicate that clos- 
ing the loop with visual feedback stabilizes the long- 
term performance of the linear algorithm (Nicolelis, 
unpublished observations). 

It is hard to relate the findings in the present study 
with these previous observations of motor cortical plas- 
ticity. In the present study, changes in neuronal coding 
were observed without concomitant signs of motor 
learning or adaptation, and the changes in coding did 
not seem to reflect a steady improvement in the coding 
of some particular movement parameter. Previous stud- 
ies have instead highlighted the relative stability of 
coding of movement direction in the absence of a 
training paradigm (Gandolfo et al., 2000). Although 
motor behavior did not change, it cannot be ruled out 
that other behavioral factors, such as attention or moti- 
vation, could change during the experimental session. A 
possible link between such factors and the observed 
drift coding of arm movement parameters in cortical 
neurons remains to be elucidated. 

Although the coding properties of single cells would 
gradually change over time, accurate predictions of arm 
position could continually be obtained from the neuro- 
nal ensemble. In other words, in spite of an individual 
neuron changing or even losing its capacity to code arm 
position from earlier to later stages of an experiment, 
other neurons would maintain or improve their capacity 
so that arm position was continuously well encoded at 
the population level. Hence, it appears that in our 
experimental situation, coding of arm position was not 
only distributed spatially over ensembles of cortical 
neurons, but also ‘‘temporally distributed’ among those 
neurons, with time constants in the order of several tens 
of minutes. Whereas adequate predictions of arm posi- 
tion could be obtained for 60 min or more, the gradual 
degradation of the performance of the linear models 
highlights the need for performance evaluation and 
model adaptation to be designed into any neuropros- 
thetic system to be used by a patient for a longer period 
of time. 


METHODS 
Surgical and Electrophysiological Procedures 


Experiments were made on two owl monkeys (Aotus 
trivirgatus). The Ethical Committee at Duke University 
approved the study. Multiple microelectrode arrays 
(NBLABS, Dennison, TX) each containing 16 or 32 
Teflon-coated, 50-um stainless steel microwires were 
implanted under general gas anesthesia with 1% iso- 
fluorane (Nicolelis, Ghazanfar, Faggin, Votaw, & Oli- 
veira, 1997; Nicolelis, Ghazanfar, Stambaugh, et al., 
1997). Stereotaxic coordinates, published microstimula- 
tion maps for owl monkeys (Preuss, Stepniewska, & 
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Kaas, 1996; Stepniewska, Preuss, & Kaas, 1993), and 
intraoperative neural mapping recordings were used to 
locate the premotor, primary motor, and posterior 
parietal cortical areas. In the first monkey (Monkey 1) 
a total of 96 microwires were implanted: 16 in the left 
primary motor cortex (MI), 16 in the left dorsal pre- 
motor cortex (PMd), 16 in the posterior parietal cortex 
(PP), 32 wires in the right PMd and MI, and 16 wires in 
the right PP cortex. In the second monkey (Monkey 2), 
32 microwires were implanted: 16 wires in the left PMd, 
and 16 in the left MI cortex. A many neuron acquisition 
processor (MNAP, Plexon Inc., Dallas, TX) was used to 
acquire and discriminate activity from single neurons 
from each implanted microwire. Time-amplitude dis- 
criminators and a modified version of a principal 
component algorithm (Nicolelis, Ghazanfar, Faggin, 
et al., 1997) were used to isolate single cortical units 
in real time. Analog samples of the action potential 
waveforms and the time of occurrence of each spike 
were stored. 


Behavioral Task 


Starting 1 to 2 weeks after surgery, animals were placed 
in a primate chair for daily recording sessions. The 
monkeys were being trained to center and move a 
manipulandum to either left or right, spontaneously or 
in response to a visual cue, to receive a juice reward. The 
position of the manipulandum was recorded continu- 
ously throughout the session using a precision potenti- 
ometer with a sampling rate of 200 Hz. All sessions were 
also videotaped. 


The Linear Regression Model 


Predictions of arm position based on simultaneously 
recorded ensembles of cortical neurons were obtained 
by applying a linear regression model to the multichan- 
nel neural data. The linear model used is an extension 
of the basic linear regression to the condition where the 
inputs (x) and outputs (y) are time series. In this case, 
significant coupling between inputs and outputs is typ- 
ically not limited to observations that are simultaneous 
in time, but may exist over some range of time delay or 
lag between the signals. In our model, X(¢) is a matrix of 
the inputs with each column corresponding to the 
discharges of individual neurons and each row repre- 
senting one time bin. For one spatial dimension, Y(¢) is 
a vector of the outputs, with samples of the position. 
The linear relationship between the neuronal discharges 
in X(¢), and arm position in Y() is expressed as 


nt 


¥(t)}=b+ > a(w)X(¢—u) +e(t) 


This equation convolves the series in X() (i.e., neuronal 
firing in time) with the functions a(z), so that the sum of 
these convolutions, plus a constant 6, approximates the 


trajectory of the hand, Y(¢). In the equation above, a(zz) 
are the weights required for fitting X@) to Y() as a 
function of time lag wu between inputs and the outputs. 
These weight functions are called impulse response 
functions. The term 6 corresponds to the y intercept in 
the regression. The final term in the equation, &(4), 
represents the residual errors, that is, any variation in 
Y(@ that cannot be explained by X@ (Brillinger, 2001, 
Bendat & Piersol, 2000). The limits of the time lag u (m 
and 7 in the above equation) should be set so that time 
lags for which statistically significant coupling exists 
between the signals in X(4) and Y(4) are included in the 
model. For a real-time application, the model has to be 
limited to positive z, that is, cortical activity in X(@ that 
has not yet occurred cannot be used to make an 
estimate of Y(f) at time ¢. Therefore, for a real-time 
application, m is zero. The optimal value for ” was 
evaluated in the present study by fitting models to the 
same sets of data with varying 7 in the equation. We also 
evaluated a modified model where a fixed time shift 0 
is introduced in the index to Y, so that neuronal activity 
is used to predict future values of Y, 


Y(?¢+0)=b+ Ss a(u)X(¢ — uw) + e(Z) 
u=0 


To fit the regression model, a temporal resolution was 
chosen, and the number of discharges of each neuron 
was counted in each bin at the desired resolution (25, 
50, or 100 msec in this study), providing X(@ in the 
equation. The model parameters a(z) and b were 
estimated using standard linear regression techniques 
(Brillinger, 2001). 


Data Analysis 


A model was first fitted to a segment of cortical and 
movement data. Data segments of 1-20 min were eval- 
uated, and 10 min was used if not otherwise indicated. 
Next, the derived model was used to predict arm 
position for another 10-min segment of data from the 
same session, and the quality of these predictions were 
used to evaluate the performance of a given model. The 
index used was the squared linear correlation coefficient 
R* between the position predictions derived from the 
model, ¥(¢), and actual recorded position, Y(t). Note 
that the R? scores presented in the present study values 
refer to predictions using new data rather than to how 
well the model fitted the data that was used for its 
derivation. 

We further analyzed in detail the impulse response 
functions a(z) for each individual neuron. Confidence 
limits for these can be calculated with a frequency- 
domain approach (Halliday et al., 1995; Rosenberg, 
Amjad, Breeze, Brillinger, & Halliday, 1989). We also 
used a statistical bootstrap method for time series: The 
original data segment used to fit the model was split into 
fragments of fixed length (5-30 sec), and the fragments 
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were independently sampled with replacement to obtain 
bootstrap samples of the original data segment. A model 
was refitted for 1000 or more such bootstrap replica- 
tions, and the distribution of the weights and other 
model parameters were evaluated to obtain confidence 
limits using standard procedures (Davison & Hinkley, 
1997; Efron & Tibshirani, 1993). We evaluated varying 
the fragment lengths and different algorithms for calcu- 
lating the bootstrap confidence limits. We only observed 
very small variations in the confidence limits. All analyses 
were done using software written by the authors in 
MATLAB (Mathworks, Natick, MA). 
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Abstract 


Brain machine interfaces (BMls) are devices that convert neural signals into commands to directly control artificial actuators, 
such as limb prostheses. Previous real-time methods applied to decoding behavioral commands from the activity of 
populations of neurons have generally relied upon linear models of neural tuning and were limited in the way they used the 
abundant statistical information contained in the movement profiles of motor tasks. Here, we propose an n-th order 
unscented Kalman filter which implements two key features: (1) use of a non-linear (quadratic) model of neural tuning 
which describes neural activity significantly better than commonly-used linear tuning models, and (2) augmentation of the 
movement state variables with a history of n-7 recent states, which improves prediction of the desired command even 
before incorporating neural activity information and allows the tuning model to capture relationships between neural 
activity and movement at multiple time offsets simultaneously. This new filter was tested in BMI experiments in which 
rhesus monkeys used their cortical activity, recorded through chronically implanted multielectrode arrays, to directly control 
computer cursors. The 10th order unscented Kalman filter outperformed the standard Kalman filter and the Wiener filter in 


both off-line reconstruction of movement trajectories and real-time, closed-loop BMI operation. 
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Introduction 


Research on brain-machine interfaces (BMI) — devices that 
directly link the brain to artificial actuators [1,2,3] — has experienced 
rapid development during the last decade primarily because of the 
expectation that such devices may eventually cure severe body 
paralysis caused by injury or neurodegenerative disease [4,5,6,7,8]. 
A core component of BMIs is the computational algorithm that 
decodes neuronal activity into commands that drive artificial 
actuators to perform movements at the operator’s will. Signal 
processing and machine learning techniques have been applied to 
the problem of inferring desired limb movements from neural 
recordings [9]. These include the population vector method 
[10,11,12,13,14,15,16], the Wiener filter [3,17,18,19,20], the 
Kalman filter [21,22,23,24], the particle filter [25,26,27,28], point 
process methods  [29,30,31,32], artificial neural networks 
[18,33,34,35], and discrete state Bayesian approaches 
[18,36,37,38]. Decoding methods using linear models of the 
relationship between neural activity and limb movements, such as 
the Wiener filter and Kalman filter, are most commonly used in 
experimental research on BMIs. These methods cannot handle 
non-linear models, which describe neuronal modulations better but 
require more complex algorithms such as the particle filter [39], a 
non-parametric recursive Bayesian estimator. However, along with 
the power of particle filters comes a heavy computational cost, 
which makes this approach difficult to implement in real-time BMI 
systems. The space of possible non-lmear models is vast, and 
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selecting an appropriate model — one that offers significant 
improvement over a linear model while avoiding “over-fitting” of 
parameters [40] — is a non-trivial task. Combined with the more 
difficult software engineering involved, these factors explain the 
rarity of non-linear models in real-tume BMI implementations. 

We propose a new computational approach for BMIs, the n-th 
order unscented Kalman filter (UKF), to improve the extraction of 
motor commands from brain activity. Our experiments showed 
that this new approach offers more accuracy compared to methods 
which use linear models while remaining computationally light 
enough for implemention in real-time. This filter offers three 
improvements upon previous designs of BMI decoding algorithms. 
First, our filter allows the use of non-linear models of neuronal 
modulations to movements (neural tuning models). Our experiments 
demonstrate the increased accuracy of our quadratic model versus 
the previously-used lmear model. Second, our filter takes 
advantage of the patterns of movements performed during the 
execution of tasks. For example, a prosthetic used to aid in feeding 
has to perform a stereotypical pattern of movements: the 
prosthetic actuator moves back and forth between the user’s 
mouth and the food items placed on a tray. Our approach uses this 
stereotypic pattern to improve BMI output accuracy. Third, our 
filter allows the relationships between neural activity and arm 
movement at multiple time offsets to be used simultaneously. 

These improvements were facilitated by extending the Kalman 
filter in two ways. First, the unscented Kalman filter [41], which 
uses a non-stochastic simulation method to approximate non- 
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linear function evaluation on random variables, was used to allow 
non-linear neural tuning models. Second, the state of our filter was 
extended to keep a history (of length n) of the desired hand 
movements to allow an autoregressive (AR n) movement model 
and neural tuning to all n consecutive time offsets. These two 
elements were combined in a system that is relatively simple, 
robust, and fast enough for real-time, closed-loop BMI applica- 
tions. 

Our algorithm was tested both off-line and in real-time, closed- 
loop experiments in which cortical recordings were obtained from 
macaque monkeys (Macaca mulatta) trained to perform two 
reaching tasks. In off-line comparisons, our method demonstrated 
significantly better accuracy compared to the Kalman filter, the 
Wiener filter, and the population vector method [10,13]. In on- 
line, closed-loop BMI control, the monkeys followed targets 
significantly better when using our method than when using the 
Kalman or the Wiener filter. 


Results 


Behavioral Tasks and Cortical Recordings 

We trained 2 rhesus macaques (Monkey C and Monkey G) to 
perform reaching tasks that incorporated stereotypic patterns of 
movements. The monkeys manipulated a hand-held joystick to 
acquire visual targets with a computer cursor (Figure 1A). In the 
center-out task, the cursor was moved from the screen center to 
targets randomly placed at a fixed radius around the center 
(Figure 1C). In the pursuit task the monkeys tracked a continuously 
moving target which followed a Lissajous curve (Figure 1D). 

Both monkeys were implanted with multielectrode arrays in 
multiple cortical areas. Monkey C was implanted in M1, PMd, 
posterior parietal cortex (PP) and supplementary motor area 
(SMA) in the right hemisphere. Monkey G was implanted 
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bilaterally in primary motor cortex (M1), primary somatosensory 
cortex (Sl) and dorsal premotor cortex (PMd). Extracellular 
discharges of 94 to 240 (average 142) cortical neurons were 
recorded while each monkey performed the behavioral tasks. 

We applied the n-th order unscented Kalman filter to the data 
collected in 16 daily sessions: 6 sessions from Monkey C and 10 
sessions from Monkey G. Data used from each session ranged from 9 
to 25 minutes. After evaluating filter accuracy off-lme, we conducted 
six on-line experiments, three with each monkey, while the monkeys 
controlled the BMI using the unscented Kalman filter and 
comparison methods in closed-loop operation. We treated the 
neurons recorded from different cortical areas as one ensemble; 
differences between individual cortical areas were not considered 
here. 


N-th Order Unscented Kalman Filter 

Our n-th order unscented Kalman filter (UKF) combined two 
extensions to the standard Kalman filter [42]: (1) the unscented 
transform [41], which allowed approximate filtering under non- 
linear models, and (2) the n-th order extension, which allowed 
autoregressive movement models and multiple temporal-offset 
neural tuning models. Figure 2 shows a comparison of the 
standard Kalman filter (Figure 2A) and the n-th order unscented 
Kalman filter (Figure 2B), as well as examples of a linear neural 
tuning model (Figure 2C), quadratic neural tuning model 
(Figure 2D), and autoregressive (AR | vs AR n) movement models 
(Figure 2D). A side-by-side comparison of the filtering equations is 
shown in Table 1. 

Like the standard Kalman filter, the n-th order unscented 
Kalman filter inferred the hidden state (the position and velocity of 
the desired movement) from the observations (neuronal rates). The 
state transition model or movement model, predicted the hidden state 
at the current time step given the state at the previous n time steps. 


A B C_Center-out task D Pursuit task 
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Figure 1. Schematics of the experimental task and cortical implants. A: The cursor and the visual targets were projected to the screen 
mounted 1.5 m in front of the monkey, and the monkeys moved the cursor with a hand held joystick with length 30 cm and maximum deflection 
12 cm. The monkeys received fruit juice rewards when they placed the cursor inside targets. B: Microwire electrode array diagram (top) and 
schematics of the placement of the arrays in the cortex of two monkeys. C: Schematics of the center-out task. After holding the cursor at the screen 
center, the monkeys moved it to a peripheral target that appeared at a random angle and a fixed radius from the center D: Schematics of the pursuit 
task. The monkeys tracked a continuously moving target whose trajectory was a Lissajous curve. 


doi:10.1371/journal.pone.0006243.g001 
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Figure 2. Comparison of the standard Kalman filter with the n-th order unscented Kalman filter. A: The standard Kalman filter predicts 
future position and velocity based on a linear model of neural tuning and predictions of the present position and velocity only. B: The n-th order 
unscented Kalman filter predicts future position and velocity based on a quadratic model of neural tuning and n history taps of position and velocity 
(AR n). C: Example of linear neural tuning model. D: Example of quadratic tuning model. E: Example AR 1 and AR n movement models. 


doi:10.1371/journal.pone.0006243.g002 


The observation model or neural tuning model predicted the expected 
neuronal rates from the estimated desired movement via a non- 
linear function. We incorporated multiple taps of the state in the 
neural tuning model to relate neural activity with hand kinematics 
at multiple time offsets simultaneously. We used a nonlinear 
quadratic model of tuning to express neuronal rates as a function of 
hand position and velocity. 


Tuning Model Validation 

We analyzed the predictive accuracy of the quadratic tuning 
model used in our n-th order unscented Kalman filter. Firing rates 
of single neurons were predicted from hand position and velocity 
using the quadratic (with n=1 and n=10 taps) and the linear 
neural tuning models after the models were fit with lmear 
regression using the Moore-Penrose pseudoinverse. A 10-fold 
cross-validation procedure was used to test predictive accuracy 
from 16 recording sessions with an average of 142 neurons 
recorded per session, and we report results using signal-to-noise 
ratios (SNR, where the signal was the recorded binned spike count) 
and correlation coefficients (CC). The n=1 tap quadratic model 
(SNR = 0.0340.29 dB, CC =0.10+0.09; mean+standard devia- 
tion) was more predictive (P<0.001, two-sided, paired sign-test) 
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than the linear model (SNR =0.0140.27 dB, CC =0.07=+0.08). 
1753 out of 2273 units (approximately 77%) were better predicted 
using the quadratic model. The n=10 tap quadratic model 
(SNR = 0.05+0.32 dB, GCC =0.11+0.10) was more predictive 
(P<0.001) than the n=1 tap quadratic model (about 900 or 
approximately 40% of units were better predicted). 

The superior performance of the quadratic tuning model is 
ulustrated in the contour plots of Figure 3, which show the tuning 
to position and velocity of eight representative neurons and 
parameter fits using the linear and quadratic (n = 1) models. The x 
and y coordinates in the plots indicate x and y positions or 
velocities and the brightness of the shading indicates the predicted 
firing rate (Figure 3, left two columns) and true firing rate (Figure 3, 
right-most column). For clarity, the fits to velocity (Figure 3, top 
four rows) and position (Figure 3, bottom four rows) are shown 
separately. The right-most column of Figure 3 shows the actual 
firing rate estimated on a 50 by 50 grid, which spanned plus and 
minus three standard deviations of the position or velocity values 
(smaller of the standard deviations for x and y) observed during the 
experimental session, using Gaussian kernel smoothing, with 
kernel width one standard deviation of the observed values 
(smaller of the standard deviations for x and y). 
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Table 1. Comparison of the equations for the standard Kalman filter and our unscented Kalman filter. 
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doi:10.1371/journal.pone.0006243.t001 


For velocity tuning (Figure 3, top four rows), the quadratic 
model captures the low-center, high-surround tuning pattern seen 
in many neurons, while the lmear model cannot capture this 
pattern because it is restricted to fitting a plane in the (rate, x, y) 
space. For position tuning (Figure 3, bottom four rows), the more 
expressive quadratic model captures the tuning patterns better 
than the linear model. While more sophisticated models of tuning, 
such as higher-order or non-parametric models, may model neural 
activity more accurately, our model is relatively simple, fast to fit 
and evaluate, and grounded in previous work (see Materials and 
Methods), while demonstrating significantly better predictive 
accuracy than the commonly-used linear model. 


Off-line reconstruction 

We compared the ability of our method to reconstruct hand 
movements from neural recordings with several commonly used, 
real-time methods by performing 10-fold cross-validation on 16 
previously recorded sessions. Parameters for the algorithms were 
fitted by ridge regression, a regularized form of linear regression, 
using recorded neural and behavioral (joystick position and 
velocity) data. The first cross-validation fold of each session was 
used to optimize ridge regression parameters and omitted from the 
results. The mean off-line reconstruction accuracy of the 10th 
order unscented Kalman filter (UKF), the Ist order unscented 
Kalman filter, the standard Kalman filter, the 10 tap Wiener filter 
fitted with ridge regression (RR), the 10 tap Wiener filter fitted 
with ordinary least squares (OLS), and the population vector 
method used by Taylor et al. are shown in Figure 4, grouped by 
monkey [13]. The y-axis shows the signal-to-noise ratio (SNR, 
where the signal was the recorded behavior) of the hand position 
reconstruction and error bars indicate plus and minus one 
standard error over the 9 cross-validation folds of each session 
and the x and y axes (for a total of 108 observations for Monkey C 
and 180 observations for Monkey G). Reconstruction accuracy for 
position and velocity, measured in SNR and _ correlation 
coefficient, for the algorithms are shown in Table 2, grouped by 
behavioral task. 
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In terms of position estimates, the 10th order UKF with our 
quadratic tuning model was consistently more accurate than the 
other algorithms. The two-sided, paired sign test with 288 
observations (16 sessions, 9 folds, 2 dimensions) and significance 
level «=0.05 was used to evaluate significance. The 10th order 
UKF produced position estimates with significantly higher SNR 
than the Ist order UKF (p< 0.001, mean difference 0.85 dB), the 
standard Kalman filter (9 <0.001, mean difference 1.25 dB), the 
10 tap Wiener filter fit using ridge regression (p<0.001, mean 
difference 1.11 dB), the 10 tap Wiener filter fit using ordinary least 
squares (p<0.001 mean difference 1.55 dB), and Taylor’s variant 
of the population vector method (p< 0.001, mean difference 5.42 
dB). When sessions of pursuit task and center-out task were 
separately analyzed, the 10th order UKF was 1.23 dB more 
accurate than the Ist order UKF in the pursuit task and 0.48 dB 
more accurate in the center-out task. 

The Ist order UKF produced position estimates with signifi- 
cantly higher SNR than the standard Kalman filter (p<0.001, 
mean difference 0.39 dB), the 10 tap Wiener filter fit using ridge 
regression (p <0.001, mean difference 0.25 dB), the 10 tap Wiener 
filter fit using ordinary least squares (p<0.001, mean difference 
0.70 dB), and Taylor’s variant of the population vector method 
(p<0.001, mean difference 4.57 dB). 

For predicting velocity, the 10th order UKF produced estimates 
with significantly higher SNR than the Ist order UKF (p <0.001, 
mean difference 0.27 dB), the standard Kalman filter (p <0.001, 
mean difference 0.36 dB), 10 tap Wiener filter fit using ridge 
regression (p <0.001, mean difference 0.29 dB), the 10 tap Wiener 
filter fit using ordinary least squares (p<0.001 mean difference 
0.82 dB), and Taylor’s variant of the population vector method 
(p<0.001, mean difference 2.60 dB). 

The Ist order UKF produced velocity estimates with signifi- 
cantly higher SNR than the standard Kalman filter (p<0.001, 
mean difference 0.09 dB), the 10 tap Wiener filter fit using 
ordinary least squares (p<0.001 mean difference 0.55 dB), and 
Taylor’s variant of the population vector method (p <0.001, mean 
difference 2.33 dB). 
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Figure 3. Contour plots of parameter fits for linear and quadratic tuning models to the tuning of eight representative neurons. The 
plot axes are the x- and y-axis of the hand position or velocity. Brighter intensity of shading indicates higher firing rate, in spikes/sec. The right-most 
column depicts the smoothed true firing rate. The quadratic model captures the trends of neuronal modulations better than the linear model for 


most neurons. 
doi:10.1371/journal.pone.0006243.g003 
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Figure 4. Off-line reconstruction accuracy for 2 monkeys (C and G) for each algorithm. Accuracy is quantified as signal-to-noise ratio (SNR) 
of the position reconstructions, averaged between x and y dimensions. Error bars indicate plus and minus one standard error. 


doi:10.1371/journal.pone.0006243.g004 


Similar results were obtained when the correlation coefficient 
was used as a measure of filter performance. 


On-line performance 

We compared the 10th order UKF to the Kalman filter and 
Wiener filter in on-line, closed-loop BMI control in six recording 
sessions: three with monkey C and three with monkey G. In each 
session, the monkey first performed the pursuit task using joystick 
control for 6 to 10 minutes. During this time period, 5 minutes of 


data was used to fit parameters for the algorithms. In each session, 
all algorithms were fit on the same data. Then the monkey 
performed the pursuit task using BMI control with each algorithm 
in turn for 5 to 8 minutes. The evaluation order of the algorithms 
was switched between sessions, however not all orderings could be 
used in the three sessions for each monkey. During BMI control, 
the monkey was required to hold the joystick as an indication of 
active participation; time periods when the monkey did not hold 
the joystick were omitted from the analysis. 


Table 2. Off-line reconstruction accuracy for the 10th order UKF, Kalman filter, Wiener filter, and population vector method. 


Filter 


Sessions 1-8 Center-out 


Sessions 9-16 Pursuit 


Mean difference from KF 


Position: SNR » CC 
10°" UKF 

1%* UKF 

KF 

WE RR 

WF OLS 

PV 

Velocity: SNR * CC 
10°" UKF 

1* UKF 

KF 

WE RR 

WF OLS 

PV 


3.24+0.16 * 0.75+0.01 
2.82+0.14 « 0.73+0.01 
2.23+0.14 « 0.71+0.01 
2.58+0.11 + 0.68+0.01 
2.29+0.11 + 0.67+0.01 
—1.31+0.13 + 0.34+0.01 


1.15+0.06 + 0.50+0.01 
1.15+0.06 + 0.48+0.01 
1.05 +0.06 + 0.47+0.01 
0.68+0.06 * 0.44+0.01 
0.42+0.06 * 0.42+0.01 
—0.51+0.08 + 0.36+0.01 


5.84+0.14 +» 0.87+0.00 
5.05+0.13 + 0.85+0.01 
3.83+0.14 * 0.83+0.01 
4.19+0.12 + 0.78+0.01 
3.97+0.11 * 0.77+0.01 
—2.24+0,23 « 0.42+0.01 


1.46+0.05 + 0.55+0.01 
1.42+0.05 * 0.52+0.01 
1.34+0.05 * 0.52+0.01 
1.10+0.04 + 0.49+0.01 
0.84+0.04 + 0.47+0.01 
—0.74+0.13 + 0.41+0.01 


1.51 + 0.05 
0.90 + 0.02 
0.00 + 0.00 
0.35 + —0.04 
0.10 + —0.05 
—4.81 « —0.39 


0.11 + 0.03 
0.09 + 0.01 
0.00 + 0.00 
—0.31-—0.03 
—0.56*—0.05 
—1.82-—0.11 


Each cell shows the SNR and CC mean=+standard error of 144 data points. The last column shows the mean difference of each algorithm compared against the Kalman 
filter, where larger numbers are better. WF RR is the 10 tap Wiener filter fitted with ridge regression. WF OLS is the 10 tap Wiener filter fitted with ordinary least squares. 


Bold numbers indicate the best value in each column. 
doi:10.1371/journal.pone.0006243.t002 
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Table 3. Comparison of behavioral performance using on- 
line, closed-loop BMI driven by a 10th order UKF, a Kalman 
filter, and a 10 tap Wiener filter fit using ridge regression. 


Session Monkey 10° UKF KF WF RR 
SNR, dB + CC 

17 Cc 2.70 + 0.69 0.70 + 0.47 NA 

18 Cc 2.73 + 0.72 2.42 + 0.60 —1.13 + 0.54 
19 Cc 2.51 + 0.71 0.80 + 0.53 0.07 * 0.68 
20 G —2.12°0.10 —1.49+°0.15 —3.23 © 0.07 
21 G 1.58 + 0.56 1.55 * 0.57 0.77 © 0.58 
22 G 3.23 - 0.71 0.39 + 0.48 —0.06 * 0.47 
Mean difference from KF 1.04-0.12 0.00 + 0.00 —1.45 + 0.00 


Performance was measured as signal-to-noise ratio and correlation coefficient 
of the BMI-controlled cursor position to the target position. The bottom row 

shows mean difference of each algorithm compared against the Kalman filter, 
where larger numbers are better. Bold numbers indicate the best value in each 
row. 

doi:10.1371/journal.pone.0006243.t003 
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Performance was measured by comparing the position of the 
target (the signal for SNR calculations) and the BMI-controlled 
cursor. Table 3 shows the signal-to-noise ratio and correlation 
coefficient for each algorithm in each session, with the mean taken 
across the x and y-axis. Figure 5 shows example traces of the BMI- 
controlled cursor and target positions in session 19. The two-sided, 
paired sign-test was used to measure significance with the two axis 
treated separately and significance value was set at =0.05. In 
terms of SNR, the monkeys performed significantly better when 
using the 10th order UKF than when using the Kalman filter 
(p<0.05, 12 observations) and 10 tap Wiener filter fitted with ridge 
regression (#<0.05, 10 observations). In terms of CC, no 
comparison was significantly different at the 2=0.05 level. 


Model, parameter, and algorithm analysis 

Our neural tuning model related neural activity with behavior 
both prior to and after the time instant of neural activity. The 
parameters past taps and future taps, in units of 100 ms, described 
the time offsets prior to and after the instant of neural activity 
between which tuning was modelled, respectively (see Materials 
and Methods). We investigated the relationship between choices of 
the number of future and past taps and reconstruction accuracy 
for the n-th order UKF (Figure 2B). The ridge regression 


Example on-line BMI position trace 


T T T T T 


T 


T 


Y position, cm 


1 1 1 1 1 


5 10 15 20 25 30 
Time, sec 


35 40 


Figure 5. Example traces of y-position during on-line, closed-loop BMI operation in a representative experimental session (session 


19, Monkey C). The dashed sinusoidal curves indicate target position. 
doi:10.1371/journal.pone.0006243.g005 
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parameter was optimized for each setting of the number of taps 
using the first fold of 10 fold cross-validation, and we report the 
accuracy on the remaining 9 folds. Plots of the mean position 
accuracy over various choices of the number of future and past 
taps for two sessions, one with center-out task (session 1) and one 
with pursuit task (session 16), are shown in Figure 6A. ‘The number 
of future taps is shown on the x-axis and each setting of past taps is 
depicted as a separate curve. For the pursuit task, the performance 
steadily increases with the number of future taps and increases 
slowly with the number of past taps. For the center-out task, the 
performance was maximum when 15 future and 2 past taps were 
used. A large number of future taps resulted in decreased 
performance, while the number of past taps had small effects on 
performance. 

To test the capacity of the movement model to predict hand 
trajectories, we conducted two analyses. In the first analysis, the 
neural tuning model update step of the 10th order UKF was 
disabled so that the filter ignored neural activity and used only the 
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movement model to “dead reckon.” In the second analysis, the 
movement model was not fit to the training data but set by 
assumption so that position was the discrete integral of velocity 
and velocity remained constant except for noise perturbations. 
The movement model noise covariance was fit to the data under 
these assumptions by calculating the mean-squared-error matrix of 
the residuals when using this movement model to predict next 
states. Figure 6B shows example traces of reconstruction under 
these two conditions on pursuit task (session 16). The true position 
of the joystick is shown by the thick dashed curve. The “dead 
reckoning” filter (dash-dotted curve) produced useless predictions 
shortly after filtering began, showing that the movement model 
could not reconstruct the hand trajectory alone, even though the 
monkey tried to follow a deterministic Lissajous curve. The 10th 
order filter with the assumed movement model (dotted curve) 
produced less accurate predictions than the filter with movement 
model fitted from the data. The position estimate SNR of the 10th 
order assumed movement model filter was 3.8840.27 dB 
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Figure 6. Dependency of reconstruction accuracy on the filter parameters. A: Reconstruction accuracy quantified as signal-to-noise ratio 
(SNR) versus number of future (x-axis) and past taps (curves). B: Example traces of position reconstruction with parts of filter disabled. The thick 
dashed curve shows the joystick x-axis position. The solid curve shows the reconstruction using the fully-functional 10th order UKF. The dotted curve 
shows the reconstruction using a 10th order UKF with the movement model assumed to be the physical equations relating position and velocity, 
instead of fitted to data. The dash-dotted curve shows the 10th order UKF with the neural observations ignored. 


doi:10.1371/journal.pone.0006243.g006 
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(mean standard error, 18 observations) for the pursuit task session 
and 2.89+0.44 dB in the center-out task session. For the fully- 
functional 10th order UKF, the SNR was 6.25+0.23 dB for the 
pursuit task session and 4.08£0.36 dB for the center-out task 
session, showing a large benefit to using a fitted movement model, 
especially for the pursuit task. The position estimate SNR of the 
Ist order assumed movement model filter was 1.57+0.76 dB for 
the pursuit task session and 0.54+0.59 dB for the center-out task 
session. Since the assumed movement model was not fitted to data 
and the movement model noise covariances were identical, this 
difference in performance between the Ist order and 10th order 
assumed movement model filters must arise from the different 
accuracies of the 1 tap and 10 tap quadratic neural tuning model. 
The large difference in accuracy (2.30 and 2.35 dB) shows the 
benefit of modeling neural tuning across multiple time offsets 
simultaneously, although much of this benefit likely comes from 
the autocorrelation of movements, which is also captured by data- 
fitted movement models. 

To quantify the extent the approximations of the unscented 
Kalman filter affected performance, we performed off-line 
reconstructions using standard particle filters with identical models 
as the Ist and 10th order unscented Kalman filter. The particle 
filters used 50,000 particles and the same parameters, initial 
conditions, and test data as the unscented Kalman filters. Since we 
had many sessions and cross-validation folds for comparison, only 
one particle filter run was performed per session and cross- 
validation fold. We used the posterior mean of the particles as the 
output. For the Ist order model, the particle filter produced 
significantly more accurate position reconstructions (two-sided, 
paired sign-test, 288 observations, p<0.001, mean difference 0.07 
dB) than the unscented Kalman filter. For the 10th order model, 
the difference in performance was not significant at the 
a%=0.05level, with the unscented Kalman filter having a nominal 
0.02 dB advantage in mean SNR. This was likely due to the large 
state space (40 dimensional) associated with the 10th order 
model—even the large number of particles could not represent 
distributions in this state space as well as a multivariate normal 
distribution, hence the UKF provided similar accuracy even with 
the unscented approximation. 


Reconstruction accuracy vs number of neurons 


Unscented Kalman Decoder 


Figure 7 shows off-line reconstruction accuracy for a pursuit 
task session when different-sized subsets of the neurons are used 
(neuron dropping curves). For each setting of the number of neurons, 
10 subsets of neurons were randomly selected and each algorithm 
was evaluated on these subsets using 10 fold cross-validation. ‘The 
first fold was reserved for finding optimal ridge regression 
parameters, and the mean accuracy on the nine remaining folds 
are plotted in Figure 7. The Ist and 10th order unscented Kalman 
filter reconstructs position more accurately than the Kalman filter, 
Wiener filter, and population vector method even for small 
numbers of neurons. The advantage of the 10th order UKF 
increases with the number of neurons. The Wiener filter fitted with 
ridge regression approaches the accuracy of the lst order UKF as 
the number of neurons increases. As expected, the benefit of ridge 
regression for fitting the Wiener filter grows larger as the number 
of neurons, and hence number of parameters, increases. Modeling 
the noise covariance between neurons becomes more important as 
the number of neurons increases, as can be seen by the lower 
performance of a modified Kalman filter which does not model 
neuron noise covariance (Kalman w/o covariance) compared to 
the unmodified Kalman filter. The neural tuning model noise 
covariance of the Kalman w/o covariance filter has all entries not 
on the diagonal set to zero. The population vector method peaks 
in performance at around 60 neurons and then decreases in 
accuracy, demonstrating the sub-optimality of the parameter 
fitting procedure which ignores covariance among neurons. 

In terms of computational load, the MATLAB implementation 
of the 10th order UKF on an Intel Pentium 4 class computer used 
0.012+0.005 seconds per iteration (mean+standard deviation), or 
around 80 Hz on average. The 30th order UKF (15 future and 15 
past taps) used 0.0360+0.0001 seconds per iteration, or around 
28 Hz on average. Our on-line implementation in C++ using 
Automatically Tuned Linear Algebra Software (ATLAS) easily 
executed faster than 10 Hz, our binning frequency. 


Discussion 


In this study, we achieved an improvement over previous 
closed-loop linear BMI decoding by implementing a more 
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Figure 7. Dependency of reconstruction accuracy for each algorithm on the number of neurons. The y-axis depicts the mean accuracy 
among 10 random subsets of neurons used by all algorithms to make reconstructions. The curve labeled Kalman w/o covariance indicates the 
reconstruction accuracy of a Kalman filter with the off-diagonal entries of the neural tuning model noise covariance set to zero. 


doi:10.1371/journal.pone.0006243.g007 
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accurate decoding algorithm, the n-th order unscented Kalman 
filter (UKF). This filter modeled arm movement profiles better 
because it used the history of past movement, and it described 
neuronal modulations to movements better by using a quadratic 
model of neuronal tuning which included tuning at multiple time 
offsets. ‘The filter performed well both in off-line reconstruction of 
previously recorded data and on-line, closed-loop BMI operation. 


Review of previous algorithms 

Much work has been done imvestigating algorithmic methods 
for decoding continuous control signals from extracellular neural 
recordings for neuroprosthetics (for a survey see Bashashati et al. 
[9]). The underlying theory stems from the pioneering work of 
Georgopoulos et al. [43], which reported the cosine relationship 
between firing rates of M1 neurons and the angle between arm 
movement and the neurons’ preferred directions. ‘The observation 
of this relationship led to a hypothesis of neuronal encoding of 
movements called the population vector model, in which 
movement velocity is calculated as vector sums of single-neuron 
vectors pointing in the neurons’ preferred directions and scaled by 
the neurons’ firing rates [10]. Many BMI studies used this 
approach to decode movement parameters from population 
activity [11,12,13,14,15]. 

The Wiener filter, an optimal linear regression method, 
improves upon the population vector approach. The Wiener filter 
has been used in many studies [3,17,18,19,20,44] and remains a 
staple of BMI research because of its relative simplicity and 
efficacy. 

As research on BMI decoding methods progressed, attention 
turned to the Kalman filter [21,22,23,24,45,46], which explicitly 
separates the models of how neural activity relates to produce 
movements and how these movements evolve over time. The 
Kalman filter, being a probabilistic method, also provides 
confidence estimates. 

Non-linear models of neural tuning provide a better description 
of neuronal modulations related to motor parameters, but are 
more computationally demanding to use. The switching Kalman 
filter, in which several Kalman filters operate in parallel using 
different parameters, was a non-linear method shown to be 
superior to the Kalman filter for BMI decoding by Wu et al. [23]. 
Another non-linear approach, called the particle filter, sequential 
Monte-Carlo, or condensation, is a recursive Bayesian estimator 
based on non-parametric representations of probability distribu- 
tions and stochastic simulation [39]. Several studies have 
investigated the particle filter for BMI decoding with a variety of 
non-linear models for neural tuning: Gao et al. [26,27], Brockwell 
et al. [25], Shoham et al. [28]. However, due to the heavy 
computational burden, online closed-loop BMI using the particle 
filter has not been reported. 

Another class of decoding methods work directly from 
individual neuron spikes instead of instantaneous firing rate 
estimates. In this approach, spike trains are modeled as discrete 
events or point processes and decoding can operate at millisecond 
time scales. The point process analog of the Kalman filter, using a 
Gaussian representation for uncertainty in state estimates and an 
inhomogenous Poisson model of spiking, was derived by Eden et 
al. (2004a, 2004b) and called the stochastic state point process filter 
(SSPPF) [29,30]. Barbieri et al. estimated the location of a foraging 
rat using recordings from CA1 hippocampal neurons and the 
SSPPF [47]. Truccolo et al. (2005, 2008) analyzed and compared 
the ability of the SSPPF to estimate several behavioral variables in 
simulations, monkeys, and humans [31,32]. Wang et al. (2006) 
showed that preserving a non-parametric posterior distribution for 
estimated hand movements using a point process particle filter 
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improves decoding accuracy versus the SSPPF in simulation [48]. 
Brockwell et al. (2007) used a Markov chain Monte-Carlo 
procedure for fitting point process filter parameters [49]. However, 
there has been no implementation of an online, closed-loop BMI 
which uses a point process filter. 

To improve decoding of simple reaching movements, tuning to 
the goal coordinates of reach trajectories has been used to 
augment tuning to movement. Kemere et al. (2004) included both 
movement tuning and target position tuning in a maximum- 
likelihood filter [50]. Srimivasan et al. (2005, 2006) incorporated 
the estimated target position of a reaching movement in both the 
Kalman and point process filter frameworks [51,52]. Later, 
Srinivasan et al. (2007) combined tuning to target position, point 
process inputs, and continuous-value inputs to allow neural spikes 
and other neural measurements such as local field potentials 
(LFPs), electrocorticography (ECoG), electroencephalography 
(EEG), and electromyography (EMG) to be used in a single 
Bayesian filter [38]. Mulliken et al. (2008) included the target 
location in the state of a Kalman filter for prediction from 
posterior parietal cortex [53]. 

Other techniques have been investigated for decoding of 
continuous hand movements. Isaacs et al. (2000) used principle 
components analysis and the nearest-neighbor algorithm [54]. Kim 
et al. (2003) proposed a competitive mixture of linear filters [55,56]. 
Sanchez et al. (2002, 2003, and 2004) and Hatsopoulos et al. (2004) 
proposed various artificial neural-network based approaches 
[18,33,34,35,57]. Shpigelman et al. (2003, 2004, and 2005) used 
support vector regression and a custom-built kernel called the 
spikernel [58,59,60]. Fisher and Black (2006) proposed an auto- 
regressive moving average (ARMA) approach [61], and Shpigelman 
et al. (2008) demonstrated the kernel autoregressive moving average 
(KARMA) method with the spikernel in closed-loop BMI [62]. In 
addition to decoding continuous hand movements, a variety of 
techniques have been employed for decoding discretized action 
choices, for example, in the studies of Hatsopoulos et al. [18], 
Musallam et al. [36], and Santhanam et al. [37]. 

While there is a large variety of algorithms available for 
decoding desired movement from neural signals, only our 
approach and the KARMA algorithm of Shpigelman et al. [62] 
have incorporated non-linear models of neural tuning in closed- 
loop BMI. 


Quadratic tuning model 

In this study, we explored whether a quadratic model of neural 
tuning can improve BMI decoding accuracy. Our analysis showed 
that our quadratic model of neural tuning was significantly more 
predictive of neuron firing rate than a linear model. We then 
implemented an unscented Kalman filter which used this 
quadratic model to infer desired hand movements. ‘The increased 
spike count prediction accuracy (0.02 dB) and off-line reconstruc- 
tion accuracy of the (1st order) UKF versus the standard Kalman 
filter (0.39 dB) and 10 tap Wiener filter (0.25 dB) demonstrates the 
benefits of our quadratic model. By using the unscented transform, 
we were able to implement a non-linear filter without resorting to 
computationally expensive particle filtering techniques. 


Movement history 

Our decoding method was further enhanced by incorporating a 
short history of hand kinematics into the hand movement model. 
We implemented an n-th order UKF which used the hand 
movement in the n previous time steps to predict hand movements 
in the next time step. Adding a short history to the state space had 
the additional benefit of modeling neural tuning across multiple 
time offsets simultaneously. When using n=10 taps, the 10th 


July 2009 | Volume 4 | Issue 7 | e6243 


159 


order UKF produced more accurate reconstruction than the Ist 
order UKF (0.85 dB improvement), demonstrating the value of 
incorporating a short history in the state space. 

We explored the optimal history length, or number of taps, for 
the UKF. Our results suggest that the number of taps for best 
performance depends on the behavioral task. For the pursuit task, 
accuracy increased with the number of future taps and plateaued 
at n= 15 or slightly later, and accuracy increased slowly with the 
number of past taps. For the center-out task, a small number of 
future taps resulted in the highest accuracy, while the number of 
past taps had small effects. The improvement of the 10th order 
UKF versus the Ist order UKF was greater for the pursuit task 
(mean 1.23 dB) than for the center-out task (mean 0.48 dB). 

Based on these results, we conjecture that the richer movement 
model of the 10th order UKF was able to capture the hand 
movement patterns produced during the performance of the 
pursuit task better than those generated during execution of the 
center-out task. This is likely because hand movements for the 
center-out task are autocorrelated over shorter time spans than 
hand movements for the pursuit task. Hand movements during 
center-out reaches were brief and unrelated between reaches, 
while during the pursuit task the hand moved relatively smoothly. 
The n taps of our movement model can be viewed as extra 
smoothing, hence the pursuit task, with smoother movement 
trajectories, benefits more than the center-out task. Our analysis 
showed that the movement model of the 10th order UKF made 
large contributions to the accuracy of the filter (6.25 vs 3.88 dB in 
pursuit task), yet this movement model was unable to provide 
accurate estimates by itself, without the aid of the neural 
recordings (dead reckoning, Figure 6B). 

In previous studies on the Kalman filter, one lingering question 
was how to set the best time offset in the model between hand 
movements and neural activity [22]. Wu et al. (2006) searched for 
the best time offset using a greedy stochastic search mechanism 
[24]. Our n-th order implementation allowed multiple time offsets 
to be used simultaneously. The ridge regression regularization 
used during parameter fitting automatically chooses the best time 
offset(s) by suppressing the weight coefficients of less useful time 
offsets. By using regularization, we have essentially replaced the 
combinatorial search for the best time offset for each neuron with 
a continuous optimization problem, at the cost of increased bias. 
We indirectly gauged the benefit of modeling tuning relationships 
across multiple time offsets by comparing the Ist and 10th order 
unscented Kalman filters with movement models assumed to be 
the physical equations relating position and velocity, instead of 
fitted to training data. The large difference in accuracy (around 
2.3 dB) showed the benefit of modeling tuning relationships across 
multiple time offsets, though much of this improvement is also 
captured by data-fitted movement models. 


Advantages of the n-th order unscented Kalman filter 

The 10-th order and 1st order UKF both produced significantly 
more accurate reconstructions than the standard Kalman filter, 
Wiener filter, and the population vector method [10,13]. In 
online, closed-loop BMI operation, the 10th order UKF allowed 
the monkey to perform a pursuit task significantly better than both 
the Kalman filter (mean improvement 1.04 dB) and Wiener filter 
(mean improvement 2.49 dB). While the SNR values reported in 
this study may seem low compared to filter performance in other 
domains, the large inherent noise in neural activity (compare 0.05 
dB mean predictive accuracy per neuron with the accuracy of 
sensors from other domains) make the BMI decoding problem 
challenging. 
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These results demonstrate the advantage of the non-linear 
model of neural tuning to arm movements at multiple time offsets 
and the advantage of leveraging patterns of movement. We have 
demonstrated one computational approach that can achieve these 
improvements without resorting to a computationally heavy 
particle filter, the filter design typically used for non-linear 
observation models. One may argue that the heavy cost of 
particle filters is not a significant obstacle due to the rapid 
improvement of computing power, for example, in personal 
desktop computers. However, an ideal BMI-driven prosthetic 
device will need to be highly mobile, placing strict limits on power 
consumption and weight, thus limiting computational power. 
While modern portable personal computers may be fast enough to 
host particle filters, they also consume dozens of watts of power 
and only manage a few hours on a typical battery pack. Thus, an 
accurate yet computationally efficient filtering algorithm is 
desirable for a compact BMI-driven prosthetic device. 

When compared to the commonly-used Wiener filter, our 
approach has several advantages. When the parameters of the 
Wiener filter are fitted using least squares, the noise of the neurons 
is assumed to be independent and of the same variance. These 
assumptions are violated by real neural populations [24]. The 
UKF explicitly models the noise of neurons in a full covariance 
matrix, allowing different variances among neurons and excess 
covariance among neurons not due to the desired output variable 
[24] to be modeled. The Wiener filter typically requires more 
parameters to be fitted than the UKF, leading to increased 
training data requirements and increased risk of overfitting. 
However, overfitting can be mitigated with regularization 
techniques such as ridge regression or sophisticated Bayesian 
regression techniques such as Variational Bayesian Least Squares 
[63]. In contrast to the Wiener filter, the UKF is a Bayesian 
technique which explicitly models the uncertainty of hand 
kinematics estimation, giving users access to measures of 
confidence in kinematic estimates. Furthermore, the UKF 
explicitly separates the neural tuning model and the movement 
model. Besides theoretical elegance, this separation allows 
parameter fitting schemes which can make better use of training 
data. For example, the model for neural tuning may be estimated 
from data obtained while the user is performing several different 
tasks, while individual movement models are estimated for each 
task. Attempting this with a Wiener filter will confound the 
autocorrelations from hand movements with the cross-correlation 
between hand movements and neural activity. 

Compared to the point process based methods, our approach 
offers less temporal resolution. However, the increased temporal 
resolution offered by point process methods comes at higher 
computational cost. The normally-distributed noise assumption 
inherant in all Kalman filters is likely violated by some neurons 
with such low firing rates that their spike counts per bin are very 
low. This is one of several approximations made for computational 
convenience in the Kalman filter approach and a main reason for 
the development of point process methods. However, point 
process methods assume all neurons are well discriminated single 
units, an assumption which is difficult to verify and which forces 
multiunits to be discarded. To model covariance of the noise 
among neurons, point process methods must model neuron 
interactions, which further increase their computational cost or 
approximation, while neuron noise covariance is included in the 
basic Kalman filter. For real-time operation on mobile devices, 
approximations and assumptions of convenience will likely be 
made by any approach, and the best algorithm will be the one 
which has the most appropriate tradeoff between accuracy and 
computational speed. 
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The switching Kalman filter proposed by Wu et al. [23] is the 
algorithm most similar to our UKF in design. The switching 
Kalman filter can be thought of as using a piecewise-linear model, 
where the pieces are combined in a weighted manner. ‘The space 
of piecewise-linear functions is clearly more expressive than the 
space of quadratic functions, but the number of pieces required to 
approximate quadratic tuning functions for each neuron over 
many input dimensions (position, velocity, history taps) is very 
large. Wu et al. reported an approximately 8.9% reduction in 
mean squared error versus the standard Kalman filter, corre- 
sponding to about 0.37 dB improvement. In comparison, our Ist 
order UKF outperforms the standard Kalman filter by about 0.39 
dB and the 10th order UKF outperforms the standard Kalman 
filter by about 1.25 dB. 

The kernel autoregressive moving average (KARMA) algorithm 
proposed by Shpigelman et al. [62] is the algorithm most similar to 
our algorithm in capability. Shpigelman et al. used a kernel 
transform custom-built for neural tuning, called the spikernel, as 
the kernel for the KARMA algorithm. This kernel allows non- 
linear, non-parametric tuning models to be used for decoding. ‘The 
KARMA algorithm, the kernel-trick extension of the well-known 
ARMA algorithm, also employs an autoregressive movement 
model to improve predictions. Like our approach, the approach by 
Shpigelman et al. has achieved real-time, closed-loop BMI 
operation with a non-linear and pattern-exploiting method. Unlike 
our approach, the KARMA algorithm is not Bayesian and does 
not directly produce confidence estimates of its output. 


Future clinical applications 

Our n-th order unscented Kalman filter is particularly suited for 
use in cortically driven prosthetic devices because of its relatively 
high accuracy and unique features. Our algorithm takes advantage 
of a non-linear model of neural tuning in a computationally 
inexpensive implementation that is well suited for mobile, low- 
power prosthetic systems. Furthermore, our algorithm takes 
advantage of patterns of movement, abundantly found in typical 
tasks such as feeding, that a prosthetic may be engaged to do. 
Since this new approach is Bayesian, it allows the computation of 
the certitude of decoded movements. Thus, decoded movements 
with low probaility can be suppressed, and undesired movements 
caused by decoding errors or unexpected neural activity can be 
detected and prevented. The separation of the neural tuning and 
movement models also allows training data to be used more 
efficiently, making the prosthetic easier to calibrate. 

The unscented Kalman filter can be applied to learn neural 
tuning model parameters or adapt to time-varying neural tuning 
and time-varying patterns of movement through a_ technique 
called dual Kalman filtering for joint parameter and hidden state 
estimation [64]. Using this approach, a person with paralysis can 
be trained to use a BMI-driven cortical prosthetic. The user first 
observes example movements performed by a technician or 
computer algorithm. Neural activity recorded from the patient’s 
brain and the example movements are then used to compute a first 
estimate of the neural tuning model. Next, the user assumes the 
control of the BMI. Then, the UKF would simultaneously decode 
neural activity and improve the estimates of the neural tuning 
model parameters. As neural tuning changes over time due to 
learning, the UKF would modifiy the neural tuning model to 
exploit these changes. Unlike the co-adaptive framework of Taylor 
et al. (2002), the UKF would compute in a _ probabilistically 
optimal fashion, without requiring knowledge of what the user is 
doing, and would update models in the background without 
explicit recalibration, making the system more user friendly. 
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The UKF can also compensate for degradation of neural 
recordings as this can be described as changes in the neural tuning 
model. Furthermore, models of movement can be improved over 
time to best predict movements produced during execution of 
particular tasks. These models can also be learned over time to 
handle novel tasks. Our future work will pursue these approaches 
toward the development of user-friendly, computationally efficient, 
and accurate algorithms for BMIs. 


Materials and Methods 


Neuronal recordings 

All surgical and experimental procedures conformed to the 
National Research Council’s Guide for the Care and Use of 
Laboratory Animals (1996) and were approved by the Duke 
University Animal Care and Use Committee. Cortical recordings 
were collected from 2 rhesus monkeys (Macaca mulatta) performing 
reaching tasks by moving a computer cursor using a hand-held 
joystick and by controlling the cursor directly through their 
cortical activity decoded by a BMI (Figure 1). Monkey C (which 
performed the task with its left hand) was implanted with four 32- 
microwire arrays in M1, PMd, PP and supplementary motor area 
(SMA) in the right hemisphere. Monkey G (which performed the 
task with its right hand) was implanted with six microelectrode 
arrays (32 microwires in each) in primary motor cortex (M1), 
primary somatosensory cortex (S1) and dorsal premotor cortex 
(PMd) of both hemispheres. Within each array, electrodes were 
grouped into 16 pairs. The separation between adjacent pairs was 
1mm. Each pair consisted of two microwires placed tightly 
together with one electrode 300 micron longer than the other. The 
longer electrode in each pair was equal or larger in diameter. 
Monkey C was implanted with stainless steel and tungsten 
electrodes of 46 and 51 micron diameter in areas SMA and M1 
and tungsten electrodes of 51 micron diameter in areas PMd and 
PP. Monkey G was implanted with stainless steel electrodes of 40 
and 63 micron diameter (Figure 1B). 

The sites with the best quality of neuronal signals were selected. 
Data from Monkey C were recorded from left PMd (9 daily 
recording sessions), left SMA (9 sessions), left M1 (9 sessions), and 
right PP (1 session). Data from Monkey G were recorded from left 
PMd (13 sessions), left M1 (13 sessions), left S1 (8 sessions), and 
right PMd (7 sessions). Extracellular neural signals were amplified, 
digitized, and high-pass filtered using Multichannel Acquisition 
Processors (Plexon, Inc.). Neuronal action potentials were 
discriminated by thresholding and sorted on-line through 
waveform templates set by the experimenter using Plexon spike- 
sorting software or using templates produced by custom-built spike 
sorting software [65]. This custom spike sorting software clusters 
waveforms by their three largest principle components using a 
modified expectation-maximization algorithm and removes spuri- 
ous clusters by thresholding on various criteria [65]. Single and 
multi-units were not treated differently for prediction purposes. 


Behavioral Tasks 

During the experimental sessions, each monkey sat in a primate 
chair. Their heads were unrestrained, and the recording system 
was connected to the implants using light flexible wires. A two 
degree of freedom (left-right and forward-backwards) analog 
joystick was mounted vertically at the monkey’s waist level. ‘The 
joystick was 30 cm in length and had a maximum deflection of 
12 cm. The monkeys were trained to manipulate the joystick with 
their hands. Monkey C performed with the left hand, and Monkey 
G performed with the right hand. An electrical resistance-based 
touch sensor on the joystick handle measured whether the monkey 
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was holding the joystick. An LCD projector projected visual 
images on a screen mounted 1.5 m in front of the monkeys 
(Figure 1A). Using the joystick, monkeys moved a round cursor, 
defined by a ring 1.6 cm in diameter. Forward, backward, 
rightward, and leftward movements of the joystick translated to 
the upward, downward, rightward, and leftward movements of the 
cursor, respectively. ‘The joystick to cursor gain varied between 3.2 
and 6.4, depending on session (i.e. a 1 cm movement of the 
joystick translated into a 3.2 to 6.4 cm movement of the cursor). 
Targets were defined by rings 16 to 20.8 cm in diameter on the 
screen. The median speeds at which monkeys moved the joystick 
were approximately 3.5 to 5.5 cm/s, depending on the session. 

Each behavioral task required placing the computer cursor over 
the target using the joystick. ‘The monkeys performed two tasks: (1) 
center-out and (2) pursuit. The center-out task (Figure 1C) used 
stationary targets that occurred at randomly chosen points on a 
fixed-radius ring around the center of the screen. The monkey had 
to hold the cursor at the center target at the screen center. After 
the center target disappeared and a peripheral target appeared, 
the monkey had to move the cursor to the peripheral target and 
keep inside the target until it received a fruit-juice reward. The 
inter-trial interval that followed a successful trial was 500 ms. The 
intertrial interval after an error trial was 700 to 1000 ms. Hold 
times varied per session from 350 to 1050 ms. The trials in which 
the monkey failed to put the cursor over the target or failed to 
fulfill the hold requirement were not rewarded. After a trial was 
finished, the center target appeared again to start the next trial. In 
our analysis, data collected during the center-out task were treated 
as a continuous stream and not segmented by trial or movement 
onset. 

The pursuit task (Figure 1D) used a moving target which 
followed a Lissajous curve: 


x= Asin(avt+6) la) 


y= Bsin(bvt) 1b) 


where x and y are the x- and y-axis coordinates and f¢ is time in 
milliseconds. We used parameter values a=3, b=4, 
ve{0.15,0.20} Hz, 6=0.52, and A=B=22.4cm (in joystick 
scale). The temporal frequency was different for the x- and y-axes, 
making the two coordinates uncorrelated. The monkey had to 
keep the cursor within the moving target to receive periodic juice 
rewards. 


Data preprocessing 

For all algorithms, spike counts were calculated in 100 ms 
nonoverlapping bins to estimate the instantaneous firing rate. 
Joystick position was recorded at 1 KHz and down-sampled to 
10 Hz to match the binning rate. Velocity was calculated from 
position by two-point digital differentiation. Position and velocity 
data were centered at their means. Spike counts were centered at 
their means for the Kalman-based filters. Data recorded while the 
monkey did not hold the joystick were disregarded. Off-line 
analysis was conducted using MATLAB (Mathworks, Inc). Real- 
time filters were implemented in a custom built BMI system 
running on a workstation with an Intel Xeon 2.2Ghz processor. 


Computational Model 


Our n-th order unscented Kalman filter (UKF) can be described 
as a modification of the Kalman filter [42], a commonly-used 
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Bayesian recursive estimation method for a specific class of hidden 
Markov models (HMMs) with continuous states and observations, 
normally distributed uncertainty, normally distributed noise, and 
linear transition and observation models (for more details. An 
introduction to the Kalman filter can be found in the Supporting 
Information section (Materials $1). The n-th order unscented 
Kalman filter combines two extensions: (1) the unscented Kalman 
filter [41], which allows arbitrary non-linear models to be used in 
Kalman filtering, (2) the n-th order extension, which allows more 
expressive autoregressive order n (AR n) movement models and 
neural tuning models. Figure 2 provides a comparison of the 
hidden Markov models for the Kalman filter (Figure 2A) and the 
n-th order unscented Kalman filter (Figure 2B). An example of a 
linear neural tuning model is shown in Figure 2C, and an example 
of a quadratic neural tuning model is shown in Figure 2D. 
Figure 2D depicts example autoregressive (AR 1 vs AR n) 
movement models (Figure 2D). 

In the hidden Markov model for BMI decoding using the n-th 
order unscented Kalman filter (Figure 2B), the hidden state is the 
position and velocity of the desired hand movement, described by 
the variable x. The state transition model or movement model, a 
linear function f,, predicts the hidden state at the current time step 
t given the state at the previous n time steps: 


x =f (x 1X 1—25++-Xt n)+Wr 1 (2) 


where W;—1 is normal, 1.1.d. noise, called the movement model noise, 
which describes the uncertainty arising from approximations made 
in the model and intrinsic randomness in the movement process. 
This movement model is an autoregressive process of order n (AR 
n), as compared to the AR 1 movement models of the Kalman 
filters previously used for BMI decoding (Figure 2D) [21,22,23,24]. 
Note that the standard unscented Kalman filter allows non-linear 
movement models, but we did not design a non-linear movement 
model and instead focused on a non-linear observation model, 
described next. 

The observation model relates the observations to the state via a 
non-linear function h: 


Vi=A( Xk Xt+k— 1 Xt k—nt 1) + (3) 


where jy; are the observations (100 ms binned spike counts) at time f¢ 
and y,; is normal, iid. noise, called the observation model noise, which 
describes the uncertainty in the neural tuning model and the intrinsic 
randomness of the neurons. The observation model predicts the 
expected neural activity for a given hand movement state. Following 
neurophysiological convention, we call it the neural tuning model. 

We incorporate multiple taps of both position and velocity in 
the neural tuning model to relate neural activity with hand 
kinematics at multiple time offsets simultaneously, avoiding the 
need to search for a best time offset [22,24]. Note that the neural 
tuning model captures relationships between neural activity at 
time f and movements from t+k—n-+ 1 up to time t+k, meaning 
that during decoding, desired movement in the future is predicted. 
We call the number k the number of future taps and n—k the 
number of past taps. In practice, the predictions into the future are 
usually inaccurate, but as they pass through the time-tap structure 
of the filter, they are improved by incorporating information from 
more neural observations. In all experiments, we used the state tap 
X; corresponding to the current observations y; as the filter output, 
Le. we did not use lagged estimates or future predictions. 
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Quadratic Neural Tuning Model 

Many models have been proposed to describe the relationship 
between neural activity and arm movement, notably the cosine 
tuning model [43], tuning to speed [31,66,67], and tuning to the 
distance of reach [68,69]. We used a more general model which 
we call the quadratic model of tuning that combined several features 
used in the previously proposed models: tuning to position, 
velocity, distance, and speed. In Cartesian coordinates, the model 
is: 


y(t) =b pos, (t) + brposy(t) +53 posy(t) + posy(t)” (4) 


+ bavel,.(t) + bs vel, (t) +b¢ vel,(t)” + vel,(t)” 


where y(f) is the mean-subtracted single-neuron firing rate at time 
t, pos,(t) and pos,(t) are the x and y coordinates of the cursor at 
time ¢, vel,(t) and vel,(t) are the x and y velocities of the cursor, 
and b;,...b6 are scalar parameters, one set per neuron. Note that 
this equation describes the quadratic neural tuning model for the 
1* order UKF. For higher values of n, additional terms for the 
other time offsets are added. For example, the 2"" order UKF with 
1 future tap and 1 past tap has a set of terms duplicated with time 
t+1. In general, our quadratic model has 6n scalar parameters per 
neuron. 

This quadratic model worked well for our experimental task in 
which the movements were performed by a joystick where the zero 
position corresponded to the center of the video screen. We chose 
not to include higher derivative terms, such as acceleration and 
jerk, because they did not contribute substantially to decoding 
accuracy. 


Implementation 

We implemented the n-th order UKF in Matlab and C++ using 
the equations presented by Julier et al. [41] with one exception: we 
used a linear movement model, which meant the first step was the 
same as that in the standard Kalman filter [42]. 

The variables in the algorithm are as follows. The vector x; of 
length 4n contained the means of the history of state variables at 
time ¢: 


pos,(t+k) 
posy(t+k) 
vel,.(t-+k) 
vel,(t+k) 
posy(t+k—1) 
posy(t+k—1) 
x,= | vel(t+k—-1) (5) 
vel,(t+k—1) 


pos,(t+k—n+1) 
pos,(t+k—n+1) 
vel, (t+k—n+1) 
vel,(t+k—n+1) 


The 4n by 47 matrix P,was the state variable covariance matrix. 
The vector y, of length V was the observed binned spike counts at 
time f, where Vis the number of neurons. 
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An iteration of the filter began with the prediction step, in which 
the state at the previous time step was used to predict the state at 
the current time step: 


Xv =F x;_1 (6) 


P,=FP,_;F7+Q (7) 


where xyand P, were the mean and covariance of the predicted 
state, X;_,and P,_; were the mean and covariance of the previous 
state, matrix F implemented the linear movement model, and Q 
was the covariance of the movement model noise. F and Q are 
square 4n by 4n matrices. Details on how these and other 
parameter matrices were fitted are described in the next section. 
Besides predicting position and velocity from previous values, the 
matrix F implemented the propagation of taps through time. 

Next, the update step corrected the prediction from the 
prediction step using the observations in a Bayesian way. In the 
Kalman filter, the neural tuning model is linear and the update 
step can be implemented in a series of matrix equations (Table 1) 
[42], because linear models allow straightforward, closed-form 
computation of the posterior distribution of the state estimate 
given the observation. However, analytical calculation of the 
posterior distribution is, in general, only possible under this linear 
model assumption [70]. For arbitrary non-linear observation 
models, computing the posterior distribution poses an intractable 
integration problem [70]. The unscented Kalman filter gives an 
approximate solution using the unscented transform — a method 
for approximating the mean and covariance of normally 
distributed random variables after they have passed through a 
non-linear function [41]. This transform uses a fixed set of 
algorithmically selected simulation points, called segma points. The 
sigma points completely capture the first and second moments of 
the distribution [70]. Geometrically speaking, the sigma points are 
located at the mean and along the eigenvectors of the covariance 
matrix, if the orthgonal matrix square root is used in their 
calculation [41], though we used the Cholesky decomposition for 
the matrix square root. 2d+1 sigma points are required, where 
d=A4n is the dimension of the state space. The set of sigma points 
is calculated from the state mean and covariance and evaluated 
through the non-linear observation function. The mean and 
covariance of the result are then calculated by taking the weighted 
mean and weighted covariance of the sigma points (for a detailed 
review see [70]). This approximation scheme computes precisely 
the effect on the mean and covariance of a normal distribution by 
the third order and below terms of the ‘Taylor expansion of the 
non-linear function, while presence of fourth order or higher terms 
in the Taylor expansion introduce error [70]. Since we use a 
quadratic observation function, the mean and covariance of our 
predicted observations are calculated precisely by the unscented 
transform. However, the non-linear observation function makes 
the distribution of the predicted observation no longer normal, 
while the unscented Kalman filtering paradigm assumes normality 
and discards the higher order moments, introducing approxima- 
tion error. Compared to the extended Kalman filter (EKF) [42], a 
well-known non-linear filtering technique, the unscented Kalman 
filter has better approximation accuracy for the same asymptotic 
computational cost [70]. 

In the general unscented Kalman filter, the sigma points are 
generated from x,;_; and P;_; and evaluated in the non-linear 
state transition and observation functions. In our implementation, 
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only the neural tuning model was non-linear, so sigma points were 
generated from x; and P;. The sigma points Xo... X2q were set 
as: 


Xo =Xy (8a) 
Xaxe+(VG+e)Pr) i=l...d (8b) 
X=xy—(/(d+0Pr) i=d+1...2d (8c) 


where the subscript outside the parentheses indicate the row taken 
from the matrix inside the parentheses. The square root is the 
matrix square root. For robustness, this computation was 
performed using the Cholesky decomposition. « is a parameter 
which specifies how heavily the center sigma point is weighted 
compared to the other sigma points. Adjusting this parameter can 
improve the approximation of higher order moments [70]. We 
used the conventional value of k=1 for normal distributions. 

Next, the sigma points were evaluated in the quadratic neural 
tuning function h: 


Zi=MX;) i=0...2d (9) 


where Zo...Zoqdenote the sigma points after observation 
function evaluation. These function evaluations were implemented 
as 2d+ lseparate matrix multiplications of the form: 


z1(t) 
z2(t) 
z3(t) | 
pos,(t) =] 

10 
fbi: Oi2 b13 bia bis bie pos,(t) oy 
bo: boa bo3 bra brs br6 pos, (t)” + posy (t)? 
b31  b32 33 b34 bss B36 vel(t) 

[ vel, (t) 
vel, (t)? +vel,(t)? | 


where 2z;(t) is the predicted (mean-subtracted) spike count for 
neuron j at time @, the vector on the left hand side is one post- 
function sigma point Z;, and the right-most vector is one pre- 
function sigma point X; with augmented terms. The bolded 
augmented terms are added to each of the sigma points using the 
sigma points’ own values for position and velocity. Note equation 
10 shows the multiplication for the 1** order UKF. For higher n, 
there are more columns of model parameters in the parameter 
matrix and more rows in the vector X; corresponding to the 
history taps. The N by 6n matrix in the center of equation 10 
containing the neural tuning model parameters, 51,1,...by.6n, for 
all V neurons is called matrix B, which has a similar function to 
matrix H of for the Kalman filter. 
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The mean and covariance of the predicted neural firing rates 
were found using weighted mean and weighted covariance: 


Zy= S- W;Zj 


i=0...2d 


(11) 


P..=Wo(Zo —21)(Zo — 21)" 


(12) 
+| © w(Zj—Zo)(Zi—Zo)" | +R 
i=1...2d 


where Ris the covariance matrix of the tuning model noise. The 
weights were: 


Wwo= (13a) 


i=1...2d 


w;= —-—~ 


d+) eh) 


Then, the Kalman gain was calculated: 


K=PP3, (14) 


where the state-observation cross-covariance P,.,.;, was: 
T 
P24 = Wo(X0 — Xr (Zo — 21) 


+ > w(Xi—Xo(Zi-—Zo)™ 
i=1...2d 


(15) 


The Kalman gain was used to correct the state estimate using 
the discrepancy between the predicted and actual (mean- 
subtracted) spike counts: 


x, =x; +K,(y,—Z) (16) 


Finally, the state covariance was updated: 


P,=P; —P..,(Pz!) PT, (17) 


Equations 6 through 17 implement one iteration of the 
algorithm. A side-by-side comparison of the equations for the 
Kalman filter and the n-th order unscented Kalman filter are 
shown in Table 1. 

In off-line reconstructions, the initial values of Xp were set by 
taking the means of the state variables in the training data, and the 
initial values of Po were set by taking the covariance of the state 
variables in the training data. When n was larger than 1, the 
means and covariances for the initial values were duplicated for 
each tap, so that the initial covariance matrix had a block-diagonal 
form with n blocks. In on-line BMI, the initial values of xo were set 
as the joystick position and velocity at that time and initially the 
values of Po were set to the identity matrix corresponding to 
variance of 1 cm for position and 10 cm/sec for velocity. 
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Parameter Fitting 

We fitted the parameter matrices F and B to training data using 
regularized linear regression and estimated the matrices Q and R 
from the regression residuals. We chose a form of Tikhonov 
regularization called ridge regression because of its simplicity and 
low computational cost. 

To fit F, we first composed the 4 by T matrix X of the training 
data position and velocities, where T is the number of data points 
(i.e. the time length of the training data). We then constructed a 4n 
by T matrix Xjagged, where column 7 of Xjaggeqd was the vertical 
concatenation of columns i— 1,i—2,...i—n of matrix X. To avoid 
the missing data problem when filling the first 2 columns of Xjagged, 
the first n columns of Xjgggeq and X were omitted when fitting F. 
Then, we fitted the intermediary matrix Fp; using ridge 
regression: 


-1 
Foart — XX iieved (KuceeaX ged ti irl) ( 18) 


where Ar was the ridge regression parameter. The selection of 
ridge regression parameters is discussed in the next section. Fpar; 
was then augmented with entries which propagated the history 
taps to make F: 


Fpart(4 x 4n) 


F= | Ta@—1)x4(n—1)) O4m—1) x4) 


(4n x 4n) 


where I is the identity matrix and 0 is the zero matrix. Subscripts 
indicate matrix sizes. 

An alternative method for setting the movement model is to use 
the equations describing motion, e.g. position is the integral of 
veloctiy over time. However, this method does not capture the 
patterns in the movements generated by the BMI user as well as 
movement models fit from kinematic data. In practice, our fits to F 
are similar to the matrix implementing the motion equations 
except for modest perturbations. 

The movement model noise covariance matrix Q was estimated 
by first computing Qyar;: 


EE} 


Qpart = (T—n)—4n 


(20) 


where Ef is the 4 by T —n residual matrix from fitting Fpa,;, and 
the division is executed per element. We then augmented Q,,,; to 
construct Q: 


Qhart(4 x4) O(4x4(n—1)) 
O(a(n—1) x an) 


(4n x 4n) 


To fit B, we first constructed the WV by T matrix Y of mean- 
subtracted binned spike counts from the training data, with the 
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spike counts from all neurons at one time step in each column. We 
then constructed the 6” by T matrix Xqug, where column 7 of Xgug 
was the vertical concatenation of columns i,i—1,....—n+1 of 
matrix X with the bolded quadratic terms in equation 10 inserted 
appropriately. To implement the / future taps, Y must be shifted 
back in time by & steps. This was done by removing the last k 
columns of Y and the first & columns of Xj,g. Subsequently, to 
avoid the missing data problem when filling the first 7 — 1 columns 
of Xqug, the first n—k—1 columns of Xgug and Y were removed. 
We then fitted B using ridge regression: 


-1 
BH YX) (XaweX e+ asl) 


aug 


(22) 


where Ag was the ridge regression parameter. 
The V by WNneural tuning model noise covariance matrix R was 
estimated using: 


E3E;, 
R= 4 _ 
(T—n+1)—6n 2) 


where Ez is the Nby T —n+1 residual matrix from fitting B, and 
the division is executed per clement. 


Algorithm Evaluation 

The n-th order unscented Kalman filter and several comparison 
methods were evaluated off-line using data collected in experi- 
ments in which monkeys moved a computer cursor using the 
joystick. The n-th order UKF used n=10 taps, with five future 
taps and fivepast taps. The UKF with n= 1 past taps was tested to 
evaluate the benefit of n=10 taps. A standard Kalman filter was 
evaluated to determine the benefit of the quadratic tuning model. 
For comparison against algorithms commonly used for a closed- 
loop BMI, a Wiener filter with 10 taps and the population vector 
method used by Taylor et al. [13] were evaluated. 

For off-line reconstructions, cross-validation was conducted. In 
this procedure, a portion of the data for each session was held-out 
for testing and the rest was used to fit parameters. Performance of 
the algorithms was evaluated on the held-out portion to avoid 
fitting models and making predictions on the same data. The data 
for each session were divided into 10 equal-sized portions (or folds) 
and the testing procedure was repeated on each held-out portion 
in turn. Both the movement and neural tuning models were fit for 
each cross-validation fold. In this study we did not address the 
question of how to design a general movement model, instead we 
leave this for future work. 

For off-line reconstructions, ridge regression parameters for every 
algorithm fitted using ridge regression were chosen by optimizing 
for highest position reconstruction accuracy on the first cross 
validation fold of each session, i.e. fitting and predicting was 
performed repeatedly for different choices of A (for the UKF, A” and 
Ag were sought independently) on the first cross validation fold. This 
first fold was omitted when aggregating performance metrics. For 
on-line experiments, ridge regression parameters were set to 
Ap =p =15for the 10 order UKF, Ap =Jg=Ay=1 for the 1* 
order UKF and Kalman filter, and 7=225 for the Wiener filter. 
These values were picked using previous experience. Wiener filter 
parameters were also fit with ordinary least squares (OLS) without 
ridge regression to demonstrate the benefit of regularization. 

The Kalman filter used for comparison had the same state 
variables as the 1* order UKF, and its models were fitted in a 
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similar way as the 1** order UKF, less the quadratic terms for the 
observation model matrix (see Supplementary Materials). 

For the population-vector method, the neuronal weights were fit 
via ordinary least squares without regularization. The original 
formulation of the population vector method predicted velocity 
and did not predict position directly. To make position 
predictions, we substituted the Cartesian position coordinates for 
the velocity components. We implemented the Taylor et al. [13] 
version of the population vector method with one slight 
modification: the baseline firing rate (mean) and normalization 
constant (standard deviation) of neurons were fit once from 
training data, instead of updated during filtering using a sliding 
window of spiking history. 

To quantify filter performance, we compared algorithm 
estimated trajectories to joystick trajectories (in off-line recon- 
structions) and to target trajectories (in closed-loop BMI). We 
computed two metrics: the signal-to-noise ratio (SNR) and the 
correlation coefficient (CC). SNR was calculated as: 


SNRap=10 logy (=) 


MSe. 


(24) 


where var is the sample variance of the desired values (joystick or 
target) and mse is the mean squared error of the predicted values 
from the desired values. Position, velocity, and the x and y axes 
were evaluated separately. The signal-to-noise ratio can be viewed 
as the inverse of the normalized mean squared error, where the 
normalization factor quantifies the power of the desired signal. 
SNR is widely used in engineering and has been previously used to 
measure BMI decoding performance [34,44]. The SNR is unitless 
and comparable across experimental setups, unlike the mean 
squared error, which is usually incomparable between studies due 
to differences in movement magnitudes. In this respect the SNR is 
similar to the CC. However, the SNR is not translation and scale 
invariant, unlike the CC. This is an advantage because translation 
and scale invariance imply that the CC may leave undetected 
certain unwanted filtering results. For example, a predicted hand 
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trajectory that is incorrect by a large but constant displacement 
has the same CC as a trajectory without the erroneous 
displacement, since only deviations from the mean are analyzed 
by the CC. As indicated by its name, CC is a measure of 
correlation, but we are interested im measuring accuracy. 
Furthermore, as the CC saturates at 1, its scale is compressed as 
it approaches 1, making it more difficult to grasp intuitively and 
making similar increments at lower values of CC and higher values 
of CC incomparable. Short of benchmark datasets, we believe the 
SNR measure best facilitates direct comparison between algo- 
rithms developed by different authors. 

To aggregate results for each session, mean SNR and CC 
among the cross-validation folds and between the x- and y-axis 
predictions were computed. Standard error of the mean was 
calculated for each session with 18 observations (9 folds x2 axes). 
To test for significant effects, we treated each cross-validation fold 
and each axis as a condition for paired, two-sided sign tests. We 
used an «=0.05 significance level. 
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ARTICLE INFO ABSTRACT 


Article history: 


While genetically modified mice have become a widely accepted tool for modeling the influence of gene 
function on the manifestation of neurological and psychiatric endophenotypes, only modest headway has 
been made in characterizing the functional circuit changes that underlie the disruption of complex behav- 
ioral processes in various models. This challenge partially arises from the fact that even simple behaviors 
require the coordination of many neural circuits vastly distributed across multiple brain areas. As such, 
Renu many independent neurophysiological alterations are likely to yield overlapping circuit disruptions and 
Chronic . ultimately lead to the manifestation of similar behavioral deficits. Here we describe the expansion of 
Mice our neurophysiological recording approach in an effort to quantify neurophysiological activity across 
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Multi-site many large scale brain circuits simultaneously in freely behaving genetically modified mice. Using this 
Neurophysiology expanded approach we were able to isolate up to 70 single neurons and record local field potential (LFP) 
Single unit activity simultaneously across 11 brain areas. Moreover, we found that these neurophysiological signals 


LFP remained viable up to 16 months after implantation. Thus, our approach provides a powerful tool that 
will aid in dissecting the central brain network changes that underlie the complex behavioral deficits 


displayed by various genetically modified mice. 


© 2010 Elsevier B.V. All rights reserved. 


1. Introduction 


Genetically modified (GM) mice have become a widely accepted 
tool for modeling the influence of gene function on behavior 
(Ekstrand et al., 2007; Giros et al., 1996; Mohn et al., 1999; Roybal 
et al., 2007; Welch et al., 2007); however, only modest headway 
has been made in characterizing the functional changes that under- 
lie the disruption of complex behavioral processes observed across 
various models. The central challenge lies within the fact that many 
of the GM mouse lines used to model central nervous system (CNS) 
disorders display alterations in genes that are expressed across 
multiple brain areas. Furthermore, even in cases where genetic 
manipulations are restricted to single brain regions, these targeted 
manipulations have been shown to induce secondary changes 
across other brain areas (Kellendonk et al., 2006). Finally, since 
even simple behaviors require the activation of neural networks 
which span multiple brain regions, many different isolated brain 
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changes are likely to be sufficient to alter circuit function and ulti- 
mately facilitate similar gross behavioral deficits. Taken together, 
these challenges present a major obstacle in identifying the specific 
genetic manipulation-induced brain changes that directly underlie 
the manifestation of a particular set of circuit deficits and ultimately 
behavioral dysfunction. 

In vivo neurophysiological recordings have become an emerg- 
ing tool for probing the functional molecular components of 
brain oscillatory networks in various GM mouse models during 
freely moving behavior (Dzirasa, 2008). For example, genetic dis- 
ruption of the 5-HT1A receptor potentiates hippocampal theta 
(4-11 Hz) oscillatory activity during anxiety related task perfor- 
mance (Gordon et al., 2005). Moreover, genetic disruption of 
the dopamine transporter (DAT) potentiates hippocampal gamma 
(30-50 Hz) oscillatory activity during exploration of a novel envi- 
ronment (Dzirasa et al., 2006), and diminishes peak hippocampal 
theta oscillatory frequencies in the home cage and during REM 
sleep (Dzirasa et al., 2009b). Knock-out (KO) mice for the gap 
junction protein Connexin-36 have also been shown to display 
impairments in hippocampal gamma oscillatory coordination (Buhl 
et al., 2003), and in an elegant gene by drug interaction study, the 
targeted activation of genetically modified M3 cholinergic recep- 
tors has been shown to induce hippocampal gamma oscillatory 
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activity (Alexander et al., 2009). Notably, in vivo recordings have 
also been utilized to investigate how selective gene manipulations 
alter neuronal firing rates and firing patterns in freely behaving 
mice (Adhikari et al., 2010; Alexander et al., 2009; Costa et al., 2006; 
McHugh et al., 2007; Zweifel et al., 2009). 

Since the induction of behavior requires the activation of neural 
circuits spanning multiple brain regions, an emerging number of 
studies have been geared towards probing circuit function in GM 
mice via in vivo neurophysiological recordings conducted simulta- 
neously across multiple brain areas. For example, it has been shown 
that acute dopamine depletion in mice lacking the dopamine 
transporter induces the synchronization of cortical-striatal ensem- 
bles (Costa et al., 2006). Moreover, studies have shown that 
mice genetically engineered to display NMDA receptor hypofunc- 
tion exhibit enhanced hippocampus-prefrontal cortex inter-area 
theta-gamma coupling, and diminished intra-area theta-gamma 
coupling (Dzirasa et al., 2009a). More recently, an elegant behav- 
ioral study demonstrated that hippocampus and prefrontal cortex 
theta oscillations synchronize during anxiety related behaviors, 
and that genetic disruption of the 5-HT1A receptor alters network 
processing across the hippocampal-prefrontal cortex pathway 
(Adhikari et al., 2010). Altogether, these studies demonstrate the 
utility of in vivo recordings in probing circuit deficits underling the 
behavioral manifestations seen in various lines of transgenic mice, 
and highlight the need for the development of tools which allow 
investigators to record more neurons per mouse and neural activity 
across entire brain circuits concurrently. 


2. Materials and methods 
2.1. Animal care and use 


Mice were separated into individual cages maintained in a 
humidity and temperature-controlled room with water available 
ad libitum, and surgically implanted with recording electrodes at 
2-5 months. Recordings were initiated following a 2-4 week recov- 
ery period. All neurophysiological recordings were conducted in an 
11” x 11” open field test environment. 

All studies were conducted with approved protocols from the 
Duke University Institutional Animal Care and Use Committee and 
were in accordance with the NIH Guidelines for the Care and Use 
of Laboratory Animals. 


2.2. Surgery 


Mice were anesthetized with ketamine (100mg/kg) and 
xylazine (10 mg/kg), placed in a stereotaxic device, and ground 
screws were secured to the anterior cranium and above cere- 
bellum. Microwires were arranged in array bundles (see Section 
3 for description of bundles), and surgically implanted based 
on stereotaxic coordinates measured from bregma. The follow- 
ing coordinates were used for experiments presented in this 
manuscript (all coordinates are measured from bregma): 


Area AP (mm) ML (mm) DV (mm) 
Ventral striatum +1.25 +1.15 —4.0 
Amygdala -1.6 -2.5 -4.8 
Fr_association_Cx +2.95 -1.5 —0.5 
Hippocampus —2.3 +1.75 1.2 
Motor cortex +1.2 —1.75 —0.5 
Prelimbic_Cx +2.5 +0.25 -0.75 
Orbital_Fr_Cx +2.6 -1.1 -1.7 
Medial dorsal thalamus -1.6 +0.3 -2.9 
Ventral posterior thalamus -1.0 -1.1 -3.3 
D.L. Striatum +0.6 —2.0 —2.0 
Substantia nigra —3.2 -1.35 —4.2 
VTA —3.2 +0.3 —4.25 


Microarray bundles targeting brain areas that were closest 
to the ground screws were implanted first. These implants were 
anchored to three screws using dental acrylic, and successive 
microwire bundles were anchored to these implants. Exposed 
microwires were covered, and electrode bases were fixed to 
microwire bundle implants using Instacure-2000 cyanoacrylate 
(Hobbylinc.com, IC-2000). For experiments with integrated elec- 
tromyographic recordings, tungsten EMG wires built directly into 
the PC boards were also placed in trapezius, triceps, and/or whisker 
muscle, and skin was closed using surgical sutures. 


2.3. Neuronal and LFP data acquisition 


Neuronal activity was recorded using the Multi-Neuron Acqui- 
sition Processor system (Plexon Inc., TX), and activity was initially 
sorted using an online sorting algorithm (Plexon Inc., TX). Only 
cells that had a clearly identified waveform with a signal-to-noise 
ratio of at least 2:1 on the oscilloscope were used. To confirm 
the quality of the recorded cells, they were sorted offline using 
an algorithm based on PCA cluster analysis (Offline Sorter, Plexon 
Inc.). All cells were presented in neuron isolation counts; however, 
only clearly defined cell clusters with clear refractory periods on 
the interspike-interval histogram (larger than 1.3 ms in this study) 
were used for PCP analysis. 

Local field potentials (LFPs) were preamplified (500x), filtered 
(0.5-400 Hz), and digitized at 500Hz using a Digital Acquisition 
card (National Instruments, Austin, TX) and a Multi-Neuron Acqui- 
sition Processor (Plexon, Dallas, TX). DC coupled headstages were 
used for neurophysiological recordings, and all recordings were 
referenced to two of the implanted screws. It is important to note 
that LFP recordings were associated with a significant phase offset 
that varied across frequencies (Nelson et al., 2008). These fixed 
phase offsets were corrected using the LFPAlign software (Plexon, 
Dallas, TX). 


2.4, Determination LFP oscillatory power and cross-structure 
synchrony 


LFPs were filtered using Butterworth bandpass filters designed 
to isolate either 1-4Hz, 4-8 Hz, 8-14Hz, 14-30Hz, or 30-55 Hz 
oscillations. The instantaneous amplitude and phase of the filtered 
LFPs were then determined using the Hilbert transform and oscilla- 
tory power within each frequency range was determined for each 
LFP by taking the mean of the amplitude envelope calculated for 
each of the filtered LFP traces. Cross-structure coherence was cal- 
culated on local field potential recording pairs using the Matlab 
mscohere function at a 2s sliding window with a 0.5s step. This 
process yielded an average coherence value, ranging from 0 to 1, 
for each brain-area pair of interest. 


2.5. Electrophysiological recording in mice treated with PCP 


Mice were treated with saline 4 weeks after surgical implanta- 
tion, and subject to recordings in a novel environment. Following 
an initial 30 min baseline recording period, mice were treated with 
PCP (3 mg/kg, i.p.) and neurophysiological data was recorded for 
an additional 30 min. The data segments recorded during the ten 
minute interval beginning 10 min after administration of saline or 
PCP was used for all analysis presented in this manuscript. In order 
to determine the effects of PCP on within-area population firing 
rates, we generated a mean firing-rate distribution for each area 
across the 10 min recording segments (30s bins) and calculated the 
99% confidence interval based on a Poisson distribution. We con- 
sidered that a population of neurons changed their firing rate if the 
mean population firing rate following PCP treatment was outside of 
the 99% confidence interval observed during the previous record- 
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Fig. 1. Floating microwire bundle implants. (A) Fixed microwire and (B) floating microwire bundle electrode designs. Each bundle is fixed to a flat cardboard or plastic 
holder to aid in surgical implantation. (C) Adult male mouse implanted with two 32 microwire array bundles. (D) Full setup can be observed, with two headstage-amplifiers 


connected to their respective microwire arrays. 


ing period (i.e., the PCP was compared to baseline saline recording 
period). 


3. Results 


In order to increase the number of neurons that we could 
acquire from each mouse, we set out double the number implanted 
microwires to 64. Our initial approach entailed implanting four 
multiwire electrode arrays (16-microwire arrays) in each mouse. 
While we have used this approach successfully in the past to 
increase the number of neurons isolated in rats and non-human 
primates (Nicolelis et al., 1997, 2003), we found that this approach 
did not work in mice. Each implanted electrode covered a cranial 
surface area of approximately 12 mm2, and electrodes required a 
minimal separation of approximately 3mm for headstages to be 
attached during neurophysiological recordings. Since the total cra- 
nial window available for implantation in mice was ~50 mm? (after 
accounting for anchoring screws), the number of electrodes (and 
thus brain areas) that we could implant per mouse was limited to 
three. To overcome this limitation, we designed novel electrodes 
where microwire bundles remained free floating from the elec- 
trode base (Fig. 1A and B). This simple design modification allowed 
us to implant microwire bundles with a minimal separation of 
<1 mm, and headstages and connectors could be fixed adjacent to 
the implanted microwire bundles on the side of the animal’s head 
(Fig. 1C and D). Since only the implanted microwire bundles were 
located directly above the cranium, up to 64 microwires arranged 
in many array bundles could be implanted in a single mouse. 
Heretofore, we describe several electrode designs and experimen- 
tal applications enabled by this approach. Importantly, with the 
exception of the simple floating design modification, the construc- 
tion and implantation of these floating bundle electrodes differed 
in no way from our previously described fixed bundle designs 
(Dzirasa, 2008). 

We created several high-density microwire array designs where 
single-wire electrodes were arranged to sample neuronal activ- 
ity throughout the thickness of brain tissue (Fig. 2A-C). Using this 
approach, we were able to implant up to 32 wires in a single brain 
area (recorded bilaterally) and isolate on average 22 neurons/area 
(Table 1). Two high-density arrays were implanted per animal for 
a total of 64 microwire channels/mouse. The total weight of the 


Table 1 
Single neurons isolated from high-density microwire implants. 


Dorsal HP (32 wires) PrL_Cx (32 wires) Total 


Mouse 1 28 27 55 
Mouse 2 21 34 55 
Mouse 3 19 15 34 
Mouse 4 19 19 38 
Mouse 5 18 25) 43 
Mouse 6 17 17 34 
Mouse 7 32 20 52 
Mouse 8 37 17 54 
Mouse 9 19 14 33 
Mouse 10 13 30 43 
Single units/wire 0.70 0.68 0.69 


implant was 2.2 g, and we found that mice could tolerate implants 
that weighed up to 10% of their free feeding body weight. Next, we 
designed high-density microwire arrays to sample brain activity 
across two brain areas simultaneously. Our initial implants tar- 
geted primary motor cortex (M1) and dorsolateral striatum (DLS) 
simultaneously using 32 microwire arrays with 16 microwires stag- 
gered in the dorsal ventral direction by approximately 1.5-1.3 mm 
(Fig. 3A and B). Using this approach, we found that we could iso- 
late on average 58 neurons/mouse (see Fig. 3C, and Table 2). This 
represented approximately twice the number of neurons that we 
were able to isolate in our previous studies (Costa et al., 2006). 
After successfully increasing the number of neurons we could 
record from a freely behaving mouse, we set out to isolate activity 
across a set of areas constituting a large-scale brain circuit. Thus, 
we targeted the four brain areas (prelimbic cortex, ventral stria- 
tum, medial dorsal thalamus, and ventral tegmental area [VTA], see 
Fig. 4A and B) comprising the dopamine reward circuit by implant- 


Table 2 
Single neurons isolated from multi-structure microwire implants. 
M1 (32 wires) Striatum (32 wires) Total 

Mouse 1 29 32 61 
Mouse 2 22 30 52 
Mouse 3 20 26 46 
Mouse 4 31 34 65 
Mouse 5 28 36 64 
Single units/wire 0.81 0.99 0.90 
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Fig. 2. High-density microwire implants. (A) Image of thirty-two 30 1m diameter microwire array bundle. (B) Schematic of high-density microwire array tips. (C) Example 
of an isolated single unit in hippocampus. From left to right: depiction of the extracellularly recorded wave forms of the unit (yellow traces); projection of the clusters 
correspondent to the unit and the noise based on analysis of the first principal component and a peak slice of the waveforms recorded and interspike-interval histogram 
showing a refractory period greater than 1.3 ms. (D) Example lesion tracks for dorsal hippocampal microwire implant. (For interpretation of the references to color in this 


figure legend, the reader is referred to the web version of the article.) 


Table 3 
Single neurons recorded simultaneously across the dopamine reward circuit. 
V. Striatum (12 wires) PrL_Cx (16 wires) Thalamus (12 wires) VTA (10 wires) Total (56) 

Mouse 1 7 17 vi 6 37 
Mouse 2 2 11 8 12 33 
Mouse 3 14 6 8 25 53 
Mouse 4 6 6 1 44 57 
Mouse 5 9 12 11 13 45 
Mouse 6 6 6 0 37 49 
Mouse 7 11 10 5 26 52 
Mouse 8 17 5 E} 48 73 
Mouse 9 2 3 E} 60 68 
Single units/wire 0.69 0.53 0.43 3.01 0.82 


ing a novel microwire array with a flexible satellite bundle design. 
Using this approach we were able to isolate the simultaneous activ- 
ity of up to 73 neurons and LFPs across the dopamine reward 
circuit simultaneously (Fig. 4C and Table 3). Next, we tested if our 
approach could be used to dissect the effect of psychoactive agents 
on neurophysiological brain networks in freely behaving mice. As 
a proof of principle, we quantified the effect of a well established 
drug of abuse, phencyclidine (PCP, 3 mg/kg i.p.), on neurophysio- 
logical activity recorded across the dopamine reward circuit and 


Table 4 


a motor circuit simultaneously. For these experiments, we used 
an additional group of mice implanted in prelimbic cortex, ven- 
tral striatum, VTA, M1, and DLS (Fig. 5A). We found that treatment 
with PCP increased 1-4 Hz (delta) oscillatory power in V. Striatum, 
and 4-8 Hz (theta) oscillatory power across all of the brain areas 
observed (Table 4). Treatment with PCP also decreased 14-30 Hz 
(beta) oscillatory power in DLS and VTA, and 30-55 Hz (gamma) 
oscillatory power in V. Striatum, DLS and M1 (Fig. 5B). Notably, 
the strongest effects of PCP on delta, theta, and gamma power 


Diverse LFP oscillatory power changes induced by treatment with PCP. Power ratio values observed in mice following PCP treatment versus those observed following treatment 


with saline; n=6 mice, *p<0.01 for all significant comparisons using paired t-test. 


1-4Hz 4-8 Hz 8-14Hz 14-30 Hz 30-55 Hz 
V. Striatum *1.42 + 0.12 "1.29 £0.11 1.12 + 0.11 0.92 + 0.04 *0.66 + 0.05 
D.L. Striatum 1.42 + 0.17 *1.29 + 0.08 1.07 + 0.08 *0.92 + 0.02 *0.85 + 0.03 
M1 1.29 + 0.40 *1.27 + 0.08 1.22 + 0.12 0.97 + 0.04 *0.94 + 0.02 
PrL_Cx 1.37 + 0.17 *1.26 + 0.10 1.31 + 0.18 0.99 + 0.03 0.99 + 0.05 
VTA 1.40 + 0.25 *1.22 + 0.08 1.07 + 0.07 *0.90 + 0.03 0.94 + 0.03 
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Fig. 3. High-density multi-area recordings in mice. (A) High-density staggered microwire array designed to target primary motor cortex (M1) and striatum simultaneously. 
(B) Schematic of multi-structure implantation strategy representing the head of the animal as seen from above and the relative position of microwire bundles (yellow) and 
their respective adaptors (red). (C) Raster plot displaying local field potentials (bottom) and 65 single neurons (top) recorded simultaneously from a freely behaving mouse. 
(D) Example of lesion tracks in M1 and striatum in an animal implanted with multi-structure electrode. (For interpretation of the references to color in this figure legend, 


the reader is referred to the web version of the article.) 


were observed in V. Striatum (~40% change in oscillatory power 
for each frequency, Tables 4 and 5). Since multiple studies have 
indicated that cross-structural neural synchrony reflects the acti- 
vation of distributed neural networks (Adhikari et al., 2010; Dzirasa 
et al., 2010b; Fell et al., 2001; Jones and Wilson, 2005; Rodriguez 
et al., 1999; Seidenbecher et al., 2003), we also quantified the effect 
of PCP on this measure. Our results indicated that the dopamine 
reward and motor circuits examined were characterized by three 
bands of high cross-structural coherence (1-6 Hz, 10-25 Hz, and 
30-55 Hz, Fig. 5C, Table 5). Moreover, we found that treatment 
with PCP enhanced 1-6 Hz coherence across the dopamine reward 
circuit (but not the motor circuit), and decreased 30-50 Hz coher- 
ence across all of the circuits examined. Treatment with PCP also 
increased the single neuron population firing rates across DLS and 
VTA, and decreased the single neuron population firing rate across 


Table 5 

Diverse dopamine reward and motor circuit changes induced by treatment with 
PCP. LFP spectral coherence ratio values observed in mice following PCP treatment 
versus those observed following treatment with saline; n=6 mice, *p<0.01 for all 
significant comparisons using paired t-test. 


1-6 Hz 10-25 Hz 30-55 Hz 
D.L. Striatum x M1 1.00 + 0.05 1.02 + 0.04 *0.78 + 0.05 
V. Striatum x Prl_Cx *1.13 + 0.04 1.03 + 0.07 *0.70 + 0.07 
PrL_Cx x VTA *1.24 + 0.06 1.05 + 0.09 *0.73 + 0.07 
VTA x V. Striatum *1.14 + 0.03 1.02 + 0.06 *0.84 + 0.06 


M1 and PrL_Cx (Fig. 5D). Together, these findings suggested that 
in principle our approach could be used as a method to dissect the 
effect of psychoactive agents on neurophysiological brain networks 
in GM mice. 

In a further step, we engineered a light weight electrode design 
to target multiple brain circuits simultaneously. For the initial elec- 
trode design, we used a total of 32 microwires to target prelimbic 
cortex, orbital frontal cortex, ventral striatum, VTA, ventral tha- 
lamus, primary motor cortex, primary sensory cortex, and DLS 
(Fig. 6A). We also modified our microwire bundle designs in order 
to target multiple brain areas simultaneously with a single bun- 
dle (Fig. 6B). Our single-electrode design reduced the weight of the 
total implant to 1.6g, allowing us to implant animals as light as 
16g. This approach may be especially suitable for younger mice, 
or genetic mutations that yield smaller mice lines. While this 
approach reduced the total number of neurons recorded per mouse 
compared to our previous designs, we were able to record neural 
activity across all of the implanted areas (see Fig. 6D, and Table 6). 
Notably, when we performed serial recordings in implanted mice, 
we found that the number of neurons isolated per mouse doubled 
between 2 and 8 weeks after implantation (Fig. 7). Incidentally, we 
also tested several electrode designs that allowed us to simultane- 
ously record electromyographic (EMG) activity (Fig. 8). 

Finally, we designed microwire electrodes to probe large-scale 
neural networks (see Fig. 9A). Using this approach, we were able 
to record neural oscillatory activity across 11 brain areas simul- 
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Fig. 4. Chronic recordings of large scale neural circuits. (A) Schematic of dopamine reward circuit. (B) Schematic of large scale circuit implantation strategy. Red circles 
represent two microwires staggered by 250 um, while black circles represent a single microwire. (C) Raster plot displaying local field potentials and 53 single neurons 
recorded simultaneously from a freely behaving mouse. (D-F) Example lesion tracks for PrL_Cx, NAC, and VTA microwire implants. Images depict the AP center of each 
implanted brain area. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.) 


taneously (ventral striatum, amygdala, frontal association cortex, 
hippocampus, M1, orbital frontal cortex, prelimbic cortex, substan- 
tia nigra, DLS, thalamus, and VTA) in freely behaving mice (Fig. 9B 
and C). Notably, the number of brain areas implanted was only 
limited by the amount of cranial surface area available for implanta- 
tion and could be increased further with modification to the design 
of individual microwire bundles. We were also able to isolate up 
to 40 single neurons/mouse across the 11 implanted brain areas 
(Fig. 9D). Since the attributes of the data obtained by our method 
were especially well suited for cross-structural neural synchrony 
studies (i.e., large number of LFPs recorded simultaneously across 


Table 6 
Single neurons isolated from thirty-two wire multi-circuit implant. 
OFC VIA M1 Acb PrL Thal $1 CP _ Total (32) 

Mouse 1 4 3 2 0 2 1 0 0 12 
Mouse 2 0 2 3 0 0 1 3 0 9 
Mouse 3 1 3 1 0 0 0 3 1 9 
Mouse 4 2 3 0 0 3 0 4 0 12 
Mouse 5 Z 3 1 0 2 0 3 0 11 
Single units/wire 0.45 0.70 0.35 O 0.35 0.10 0.65 0.05 0.66 


many brain areas), we used coherence analysis to examine the 
cross-structural synchrony profiles of the 11 implanted brain areas. 
Surprisingly, our results indicated that cross-structural neural syn- 
chrony patterns differed widely across the brain area pairs used for 
coherence analysis (Fig. 10). For example, amygdala tended to syn- 
chronize with ventral striatum and VTA in the gamma frequency 
range, while hippocampus tended to synchronize with amygdala 
and thalamus in the theta frequency range. Additionally, some 
brain areas pairs displayed low overall synchrony (i.e., thalamus 
and frontal association cortex), while others displayed high syn- 
chrony across all frequencies (i.e., thalamus and substantia nigra). 
Together these results demonstrated that our approach was capa- 
ble of characterizing the dynamic neural interactions which define 
widely distributed brain circuits. When we examined the stability 
of our neural and neuronal signals across each of the 11 brain areas, 
we found that neural signals were still viable more than a year after 
electrode implantation (n =2 mice, Fig. 11). Moreover, we were able 
to isolate neuronal signals more than a year after implantation as 
well (Table 7). Importantly, these results suggest that neurophys- 
iological signals recorded using our recording approach are likely 
to be viable for the entire lifespan of implanted mice. 
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Fig. 5. Diverse dopamine reward and motor circuit changes induced by treatment with PCP. (A) LFP and single units recorded in the same session following treatment with 
saline and PCP. (B) Power spectral plots calculated from LFPs recorded in mice treated with Saline and PCP. (C) LFP spectral coherence plots generated for LFPs recorded 
across the dopamine reward and motor circuit. Mice displayed a 1-6 Hz, 10-25 Hz and 30-55 Hz coherence band across the circuits examined. (D) Treatment with PCP 
increased the DLS and VTA population firing rate, and reduced the M1 and PrL-_Cx population firing rate; p <0.01 for all significant comparisons. Green highlights correspond 
with significant increases, while red highlights correspond with significant decreases in the population firing rate. (For interpretation of the references to color in this figure 


legend, the reader is referred to the web version of the article.) 
4. Discussion 


GM mice have become a widely accepted tool for modeling the 
influence of gene function on behavior (Ekstrand et al., 2007; Giros 
et al., 1996; Mohn et al., 1999; Roybal et al., 2007; Welch et al., 
2007); however, only modest headway has been made in charac- 
terizing the functional brain changes that underlie the disruption 
of complex behavioral processes observed across various models. 
Since these behavioral deficits ultimately result from changes in 
neural circuits which span multiple brain regions, we have created 
an approach that allows brain activity to be probed across mul- 
tiple large-scale brain circuits simultaneously in freely behaving 
GM mouse models of neuropsychiatric illness. Our findings indicate 


Table 7 
Single neurons isolated in freely behaving mice more than a year after surgical 
implantation. 


1 month 13-16 months 
Mouse 1 27 15 
Mouse 2 39 30 


that our expanded neurophysiological recording approach is capa- 
ble of isolating up to 70 single neurons in a single freely behaving 
mouse. This represents a 2-3 fold increase over the number of single 
neurons previously isolated in freely behaving mice using our clas- 
sic neurophysiological recording techniques. We also demonstrate 
that our technique can be used to acquire local field potential and 
unit activity across 11 brain areas simultaneously in freely behav- 
ing mice, though this approach yields a diminished number of units 
that can be isolated from each implanted brain area. Local field 
potentials, on the other hand, can be isolated from each recording 
wire implanted in each brain area yielding a maximum total of 64 
continuous signals/mouse. Moreover, we show that these signals 
remain viable for over 12 months after implantation. 

LFPs are classically considered inferior neurophysiological sig- 
nals compared to single units since LFPs are subject to signal 
artifacts that may result from gross volume conduction, electro- 
physiological noise introduced from referencing sites (i.e., ground 
screws, 60Hz artifact), or background noise generated during 
behaviors such as chewing, yawning, and grooming (not to men- 
tion others). However, our multi-circuit recording approach has 
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Fig. 6. Lightweight multi-circuit recordings in mice. (A) Schematic of multi-circuit implantation strategy using one 32 microwire electrode. (B) Microwire bundle designed 
to target three brain structures simultaneously. (C) Mouse implanted across eight areas with a 32 microwire electrode. (D) Raster plot displaying local field potentials and 
29 single neurons recorded simultaneously across eight brain areas in a freely behaving mouse. 


the potential to overcome many of these limitations. First, electro- 
physiological and background noise would vary equivalently across 
time in all of the simultaneously recorded LFP signals. Thus, algo- 
rithms can be developed to isolate these aberrant signals and filter 
them appropriately across channels. Second, one can determine 
if recorded LFP signals result from pertinent neurophysiologi- 
cal interactions or simply volume conduction since our approach 
allows neurophysiological activity to be acquired simultaneously 
across brain areas located throughout the brain (various depths and 
rostral—-caudal locations). Thus, LFP signals that result from volume 
conduction will generate spectral synchrony between LFP pairs that 
is also observed in LFP signals recorded intermediate to the two LFP 
pairs of interest. An example of this approach is demonstrated in 
our recent study where we showed that acute depletion of nore- 
pinephrine differentially affects orbital frontal cortex—striatal delta 
coherence and frontal association cortex-striatal delta coherence 
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Fig. 7. Neurons isolated across multiple brain circuits during serial recordings in 
mice. 


despite the fact that the cortical electrodes were located ~1 mm 
apart (Dzirasa et al., 2010a). Finally, our findings also provide 
evidence that our approach can overcome limitations in analy- 
sis that result from behavioral noise (i.e, movement artifacts). 
For example, our findings demonstrated that treatment with PCP 
differentially affects motor and reward circuits (as measured via 
inter-area coherence), despite the increased behavioral profiles 
induced by treatment with the PCP. While movement artifacts are 
not necessarily equally present across all LFP channels, these signals 
vary equivalently across LFP channels across time and ultimately 
generate spectral coherence that varies equivalently and not dif- 
ferentially across LFP pairs. 

One of the most notable findings observed using our multi- 
circuit (>10 simultaneously recorded brain areas) approach is 
that different brain areas tend to synchronize in different fre- 
quency ranges. For example, V. Striatum tends to synchronize with 
amygdala and V. Striatum in the gamma frequency range, while 
hippocampus tends to synchronize with amygdala and thalamus 
in the theta frequency range. Thus our approach provides a power- 
ful tool that can be potentially used to dissect the neural oscillatory 
modes underling circuit level interactions across specific brain net- 
works. 

Multiple in vivo electrophysiological studies have been con- 
ducted in rodents in an effort to study the effect of drugs of abuse on 
brain function. For example, it has been shown that cocaine-seeking 
behavior is encoded in the core and shell of nucleus accum- 
bens (Carelli et al., 2000; Owesson-White et al., 2009). Moreover, 
repeated cocaine exposure has been shown to abolish the pres- 
ence of membrane bi-stability normally present in neurons located 
in the limbic prefrontal cortex (Trantham et al., 2002). Neverthe- 
less, the approaches used in these studies were severely limited 
in their ability to identify and characterize the dynamic brain net- 
work changes that underlie the manifestation of addiction-related 
behaviors since they only acquired brain activity across a small 
number of brain areas at a time. Our findings observed in mice 
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Fig. 8. Simultaneous multi-structure and electromyographic recordings in mice. 


treated with PCP provide evidence that drugs of abuse induce a 
wide array of changes that vary across brain circuits. For exam- 
ple, our results showed that treatment with PCP increased the 
population firing rate of DLS and VTA neurons, and decreased the 
population firing rate of M1 and PrL_Cx neurons. On the other hand, 
treatment with PCP had no effect on the population firing rate 
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Fig. 9. Chronic multi-cell multi-circuit recordings in mice. (A) Schematic of multi-circuit implantation strategy using two 32 microwire electrodes. (B) Schematic of multi- 


of V. Striatal neurons. Treatment with PCP also induced differen- 
tial changes at the level of circuit synchrony. For example, PCP 
enhanced 1-6 Hz spectral coherence across the entire dopamine 
reward circuit, but not the motor circuit, and reduced gamma spec- 
tral coherence across both of the circuits examined. Together, this 
demonstrates that drugs of abuse induce a diverse array of cir- 
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circuit recordings across 11 brain areas. (C) Local field potentials and (D) forty single neurons recorded simultaneously across 11 brain areas. 
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Fig. 10. Synchrony across large scale neural circuits in mice. Coherence plots calculated for brain area pairs depicted in Fig. 6B. Each coherence plot represents data recorded 
simultaneously in a freely behaving mouse during a 5-min session in its home cage. High neural synchrony was observed between thalamus and substantia nigra, while low 
neural synchrony was observed between dorsal striatum and substantia nigra. 
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Fig. 11. Long-term neural recordings in mice. Local field potentials recorded in the same mouse 1 month (left) and 16 months (right) after surgical implantation. 


46 K. Dzirasa et al. / Journal of Neuroscience Methods 195 (2011) 36-46 


cuit level changes across different brain networks. Notably, since 
neurophysiological signals can be acquired in freely behaving mice 
more than a year after their initial surgical implantation, our neuro- 
physiological recording approach serves as a powerful tool that can 
be used to simultaneously examine multiple circuit level changes 
underlying the acquisition, expression, and extinction of addiction 
related behaviors. Several GM mouse line have been shown to dis- 
play altered behavioral responses to drugs of abuse. For example, 
mice with an N-ethyl-N-nitrosurea induced point mutation in the 
circadian gene Clock (Clock-A19 mice) have been shown to dis- 
play increased sensitivity to rewarding stimuli including cocaine 
(Roybal et al., 2007), while mice lacking a phosphoprotein down- 
stream of D1 and D2 dopamine receptors (DARPP-32 KO mice) 
have been shown to display a diminished behavioral response to 
amphetamine, PCP, and LSD (Svenningsson et al., 2003). Thus, our 
approach provides a powerful tool that can be used with these 
GM models to dissect how changes in gene and protein expression 
alter neurophysiological circuits responsible for the acquisition and 
expression of addition related behaviors. 


5. Conclusion 


Overall, our chronic neurophysiological recording approach 
provides a powerful tool for dissecting the circuit changes under- 
lying behavioral changes seen various GM mouse models of 
neurological and psychiatric diseases and probing the effect of 
pharmacological agents on these circuits. Moreover, our recording 
approach can potentially be integrated with novel optogenetic and 
chemicogenetic tools capable of manipulating neuronal activity in 
order to quantify the role that specific cell types play in modulat- 
ing neurophysiological activity across large scale circuits spanning 
multiple brain areas. 
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Brain-machine interfaces (BMIs) transform the activity of neurons 
recorded in motor areas of the brain into movements of external actu- 
ators. Representation of movements by neuronal populations varies over 
time, during both voluntary limb movements and movements controlled 
through BMIs, due to motor learning, neuronal plasticity, and instability 
in recordings. To ensure accurate BMI performance over long time spans, 
BMI decoders must adapt to these changes. We propose the Bayesian 
regression self-training method for updating the parameters of an un- 
scented Kalman filter decoder. This novel paradigm uses the decoder’s 
output to periodically update its neuronal tuning model in a Bayesian 
linear regression. We use two previously known statistical formulations 
of Bayesian linear regression: a joint formulation, which allows fast and 
exact inference, and a factorized formulation, which allows the addition 
and temporary omission of neurons from updates but requires approx- 
imate variational inference. To evaluate these methods, we performed 
offline reconstructions and closed-loop experiments with rhesus mon- 
keys implanted cortically with microwire electrodes. Offline reconstruc- 
tions used data recorded in areas M1, S1, PMd, SMA, and PP of three 
monkeys while they controlled a cursor using a handheld joystick. The 
Bayesian regression self-training updates significantly improved the ac- 
curacy of offline reconstructions compared to the same decoder without 
updates. We performed 11 sessions of real-time, closed-loop experiments 


Neural Computation 23, 3162-3204 (2011) © 2011 Massachusetts Institute of Technology 


179 


Bayesian Adaptive Decoding for Brain-Machine Interfaces 3163 


with a monkey implanted in areas M1 and S1. These sessions spanned 29 
days. The monkey controlled the cursor using the decoder with and with- 
out updates. The updates maintained control accuracy and did not require 
information about monkey hand movements, assumptions about desired 
movements, or knowledge of the intended movement goals as training 
signals. These results indicate that Bayesian regression self-training can 
maintain BMI control accuracy over long periods, making clinical neuro- 
prosthetics more viable. 


1 Introduction 


Brain-machine interfaces (BMI) are systems that directly connect brains to 
external devices, for example, prosthetic limbs for persons with paralysis 
(Nicolelis, 2003). While BMI systems have to perform accurately over long 
time periods, changes in the properties of recorded neuronal population 
may cause significant alterations in BMI operation. Studies have shown that 
neuronal tuning to arm movements may change over time spans as short as 
1 hour (Li, Padoa-Schioppa, & Bizzi, 2001; Padoa-Schioppa, Li, & Bizzi, 2004; 
Lebedev et al., 2005; Kim et al., 2006; Rokni, Richardson, Bizzi, & Seung, 
2007; Chestek et al., 2007). Changes in neuronal tuning over time have been 
also reported for BMI control (Taylor, Tillery, & Schwartz, 2002; Truccolo, 
Friehs, Donoghue, & Hochberg, 2008). Additionally, neuronal and recording 
instability can cause changes to the shape of recorded spike waveforms. 
These changes may cause the spike-sorting system to err and lead to changes 
in the effective tuning of recorded units. For these reasons, BMI systems 
need adaptive decoding to maintain control accuracy over long time periods 
(Kim et al., 2006; Wu and Hatsopoulos, 2008). In addition to the ability to 
track neuronal variability, adaptive decoding has the benefit of exploiting 
plastic changes in neuronal tuning that occur during learning and thus 
facilitate coadaptive (Taylor et al., 2002) synergies between the user’s brain 
and the BMI. 

Early computational methods for translating recorded neuronal sig- 
nals into movement commands assumed static neuronal tuning (Wessberg 
et al., 2000; Serruya, Hatsopoulos, Paninski, Fellows, & Donoghue , 2002; 
Carmena et al., 2003; Hochberg et al., 2006; Wu, Gao, Bienenstock, 
Donoghue, & Black, 2006; Ganguly & Carmena, 2009). These methods fixed 
the parameters of the neuronal decoder after an initial period of algorith- 
mic training or parameter fitting. Such decoders required users to adapt to 
the BMI control system by changing their neuronal modulations, and these 
changes were not tracked by the decoder. 

Some studies have implemented adaptive neuronal decoders (Sykacek & 
Roberts, 2002; Eden, Frank, Barbieri, Solo, & Brown, 2004; Rickert, Braun, & 
Mehring, 2007; Srinivasan, Eden, Mitter, & Brown, 2007; Wu & Hatsopoulos, 
2008; Shpigelman, Lalazar, & Vaadia, 2009; Gilja et al., 2009; Liao et al., 
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2009). However, most of these neuronal decoders have assumed that the 
desired movements—for example, the trajectory from the initial position of 
the actuator to a target—were available to the update method. The update 
method then adjusted the decoder parameters such that the BMI output 
more closely approximated the desired movement. One approach to ensure 
the availability of desired movements is to instruct the prosthetic user to 
attempt prescribed movements during periodic retraining sessions. 

A more versatile approach to decoder update is to shift the paradigm 
from assuming a priori knowledge of desired movements to using a com- 
putational method that updates the decoder’s model automatically, with 
little or no external intervention. The goal of this study is to design and 
validate an update method that can achieve this objective. 

We propose the Bayesian regression self-training method for adaptive 
decoding. This method periodically updates (every 2 minutes in our study) 
the parameters of the model of neuronal tuning while decoding user inten- 
tions. During each update, the procedure combines information from previ- 
ous model parameters, recent neuronal activity, and outputs of the decoder 
to compute new model parameters. Instead of using a priori knowledge of 
the desired movements, the updates use the decoder outputs as substitutes 
for the desired movements (similar to Gilja et al., 2009; Liao et al., 2009). 
Bayesian linear regression (Bishop, 2006; Rencher & Schaalje, 2008) between 
decoder outputs and neuronal activity, with the previous parameters acting 
as priors, calculates the new parameters. We chose Bayesian linear regres- 
sion because it combines previous parameters and tuning changes detected 
from recent neuronal activity in a probabilistic manner, so that the new 
parameters are the posteriors of a Bayesian update. We also formulate a 
transition model for the tuning parameters. Together, the Bayesian regres- 
sion update and transition model form a batch-mode Bayesian filter for 
the tuning parameters. For the decoding algorithm, we use the nth order 
unscented Kalman filter (Li et al., 2009). Our decoder and parameter up- 
date system simultaneously predicts the BMI user’s desired movements 
and tracks tuning changes in the user’s neuronal modulation. 

Unlike previous, nonprobabilistic update approaches (Wu & Hatsopou- 
los, 2008; Shpigelman et al., 2009; Gilja et al., 2009), our approach auto- 
matically takes into account the uncertainties of the tuning model param- 
eters and the amount of useful information and noise in the new data to 
compute Bayesian updates of tuning parameters. Unlike previous studies 
with parameter updates at every decoding time step (Eden et al., 2004; 
Rickert et al., 2007; Srinivasan et al., 2007; Wu & Hatsopoulos, 2008), our 
batch-mode approach allows the use of the Kalman smoother (Rauch, Tung, 
& Striebel, 1965) to improve decoder predictions before their use in updates. 
The Kalman smoother uses the model of movement change over time to 
refine predictions of desired movements at a time t using predictions of de- 
sired movements at times after t. The noncausal nature of this preprocessing 
step means it is available only in batch-mode operations. 
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We tested our method with rhesus monkeys that performed motor tasks 
manually or through a BMI. We started with offline reconstructions of 
hand trajectories from neuronal data previously recorded in three monkeys. 
We then conducted experiments in which one monkey used our adaptive 
method ina real-time, closed-loop BMI. Adapting within each session in the 
offline reconstructions, our decoder was more accurate than the equivalent 
unscented Kalman filter without updates. Adapting within and across ses- 
sions in the closed-loop experiments, our adaptive decoder maintained BMI 
control accuracy over 29 days, while the unscented Kalman filter without 
updates degraded in accuracy. These results indicate that Bayesian regres- 
sion self-training can cope with changes in neuronal tuning or neuronal 
recording without the need for example movements. Bayesian regression 
self-training improves the robustness of a BMI to changes in neuronal tun- 
ing, making long-term BMI operation more reliable. 


2 Neurophysiological Methods 


2.1 Neuronal Recordings. We conducted experiments in three macaque 
monkeys (Macaca mulatta) implanted with microwire arrays in multiple arm 
representation areas of the cortex. All surgical and experimental procedures 
conformed to the National Research Council’s Guide for the Care and Use 
of Laboratory Animals (1996) and were approved by the Duke University 
Animal Care and Use Committee. We trained the monkeys to move a cursor 
on a computer screen either using a handheld joystick or through a BMI 
that generated cursor position from neuronal activity. 

Figure 1A shows a diagram of electrode geometries and array placement 
for monkeys C, G, and M. All implants were custom-made by engineers 
from our lab (Lehew & Nicolelis, 2008). Monkeys C and G had similar 
implants. Monkey C was implanted with four 32-microwire arrays in the 
primary motor cortex (M1), dorsal premotor cortex (PMd), posterior parietal 
cortex (PP), and supplementary motor area (SMA) in the right hemisphere. 
Monkey G was implanted with six 32-microwire electrode arrays in M1, 
primary somatosensory cortex (S1), and PMd of both hemispheres. Elec- 
trodes in each array were grouped into pairs and placed on a uniform grid, 
with a 1 mm separation between pairs. The electrodes in each pair were 
placed tightly together, and one electrode was 300 microns longer than 
the other. The recording site on each electrode was at the tip. The material 
and diameter of the electrodes varied, and the longer electrode had equal or 
greater diameter. Monkey C was implanted with stainless steel and tungsten 
electrodes with diameters 46 and 51 microns, respectively, in areas SMA and 
M1 and tungsten electrodes with a diameter of 51 microns in areas PMd 
and PP. Monkey G was implanted with stainless steel electrodes measuring 
40 and 63 microns in diameter. 

Monkey M was implanted with four 96-microwire arrays in M1 and 
S1 bilaterally in arm and leg regions so that one array corresponded to the 
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Figure 1: Implants and experimental methods. (A) Implant placement and ge- 
ometries. Microwire diameters are specified in microns. To reduce clutter, we 
show only arrays from which data were used. (B) Primate experiment setup. 
(C) Tuning model update paradigm. Updates occur periodically using the neu- 
ronal data and decoder outputs generated since the previous update. (D) Three 
behavioral tasks. Each task required placing the cursor over a target. In center- 
out, targets were stationary. In the pursuit tasks, targets moved continuously 
according to a Lissajous curve or a set of invisible, random waypoints. 


cortical regions associated with each limb. In this study, we used recordings 
only from the right hemisphere arm array, since monkey M manipulated the 
joystick with its left hand. Within each array for monkey M, microwires were 
grouped in two 4 x 4 uniformly spaced grids of 16 triplets of electrodes. 
One grid was placed over M1, and one grid was placed over S51. The grids 
were separated by 2 mm, which bridged the gap of the central sulcus. The 
separation between triplets of electrodes within each grid was 1 mm. The 
electrodes of each triplet had three different lengths, staggered at 300 micron 
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intervals, and recorded from the tips. The penetration depth of each triplet 
could be adjusted as the triplets could be moved individually in the depth 
axis, but the depth separation of the triplets was fixed. Electrodes were 
inserted into the brain 1 to 2 weeks after surgery (during which the array 
base was implanted on the surface of the brain). Data used in this study 
were collected while the longest electrodes in all triplets were set to 2 mm 
in length. The electrodes were constructed of stainless steel. The electrode 
diameters in the array were heterogeneous, with each microwire having a 
diameter of 50 to 63 microns. 

To extract motor parameters from neuronal activity, we sampled data 
from cortical sites contralateral to the arm that held the joystick. Data from 
monkey C (total 5 sessions) were recorded from M1 in 5 daily experimental 
sessions, PMd in 5 sessions, SMA in 5 sessions, and PP in 1 session. Data 
from monkey G (total 8 sessions) were recorded from left M1 in 8 sessions, 
left S1 in 7 sessions, left PMd in 8 sessions, and right PMd in 5 sessions. Data 
from monkey M (total 16 sessions) were recorded from right M1 and right 
S1 in all 16 sessions. We included neurons from S1 for predictions because 
prior work has found that they can be used for decoding arm movements 
(Carmena et al., 2003). For offline reconstructions of hand movements using 
neuronal data, we used 18 experimental sessions (monkey C: 5 sessions, G: 8 
sessions, M: sessions), each with 94 to 240 (average 143) recorded neurons. 
We conducted 11 closed-loop BMI control sessions (each recording 139 
neurons) with monkey M. Extracellular neuronal signals were amplified, 
digitized, and bandpass filtered using Multichannel Acquisition Processors 
(Plexon, Inc.). Neuronal action potentials were triggered using thresholds 
and sorted online by the Multichannel Acquisition Processors, which ran a 
template matching algorithm for neuronal waveform templates set by the 
experimenter. 


2.2 Behavioral Tasks. In experimental sessions, monkeys sat in a pri- 
mate chair in front of a large screen, mounted 1.5 m in front of them (see 
Figure 1B). The recording system was connected to the implants using light, 
flexible wires, as the monkeys’ heads were not restrained. An analog joystick 
with 2 degrees of freedom (left-right and forward-backward) was mounted 
vertically so that the monkeys grasped the joystick tip with their hands. 
The joystick was 30 cm in length and allowed a maximum linear deflection 
of 12 cm. Monkey C and monkey M controlled the joystick with their left 
hands, and monkey G used its right hand. An electrical resistance touch 
sensor or an optical touch sensor on the joystick tip determined whether 
the monkeys were holding the joystick. An LCD projector, controlled by a 
computer with custom-built BMI experimental software, projected white 
visual stimuli on a black background on to the screen. 

During hand control, monkeys moved a cursor (a ring 1.6 cm in diame- 
ter) using the joystick. The joystick to cursor gain varied between 3.2X and 
6.4X, depending on session. Targets appeared as rings 16 cm to 20.8 cm 
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in diameter. The median speeds at which monkeys C and G moved 
the joystick were approximately 3.5 cm/s to 5.5 cm/s, depending on the 
session. The median speed at which monkey M moved the joystick was 
approximately 0.5 cm/s to 1.1 cm/s, depending on the session. The slower 
movements of monkeys M were due to lower target speeds in the behav- 
ioral task. We set the target speeds lower because we found that slower 
movements were decoded more accurately, which is important for model 
updates when decoded trajectories are used for the training signal. All ani- 
mals were overtrained (at least 1 year of experience) in performing the tasks 
with a joystick. Monkey M had 5 months of experience with BMI control 
before the closed-loop sessions. 

During BMI control, we decoded the monkey’s neuronal modulations 
using an unscented Kalman filter (Li et al., 2009) to control the cursor. We 
fitted the unscented Kalman filter’s parameters using data recorded dur- 
ing hand control and updated the filter’s parameters using the Bayesian 
regression update method (experimental condition) or left them fixed (con- 
trol condition). The monkey had to grasp the joystick to signal participation 
(otherwise the task paused and stimuli were not shown), but joystick move- 
ments were not used for cursor control. 

The behavioral tasks required the monkeys to place the cursor over 
the target, using either the joystick or BMI. We analyzed data from three 
behavioral tasks: center-out, pursuit with Lissajous trajectories, and pursuit 
with point-to-point trajectories (see Figure 1D). 

In the center-out task (see Figure 1D, top), the monkeys had to hold the 
cursor in a stationary center target and then move to a peripheral target 
randomly placed on a fixed-radius circle around the center target. After 
holding at the peripheral target and receiving a juice reward, a new trial 
began with the disappearance of the peripheral target and the appearance 
of the center target. The intertrial interval that followed a successful trial 
was 500 ms, and the intertrial interval after a failed trial was 700 to 1000 ms. 
Hold times varied per session from 350 to 1050 ms. We treated data collected 
from this and all other tasks as a continuous stream and did not segment 
by trial or movement onset. 

In the pursuit task with Lissajous trajectories (see Figure 1D, middle), a 
continuously moving target followed a Lissajous curve: 


x(t) =A- sin(avt + 4), (2.1a) 
y(t) =B- sin(but). (2.1b) 


The variables x and y are the x- and y-axis coordinates, and t is time in 
seconds. We used parameters a = 3,b = 4,5 = 0.57, v € {0.1, 0.15, 0.2} Hz, 
and A = B = 22.4 cm (in screen scale). The x- and y-axis coordinates were 
uncorrelated because the temporal frequency was different for each. To 
receive periodic juice rewards, the monkeys had to keep the cursor over 
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the moving target. Rewards were larger when the cursor was closer to the 
center of the target to motivate the monkeys to pursue the target accurately. 
This was done by scaling the amount of juice per reward (given every 750 
ms of holding inside the target) by the cube of the difference of target radius 
and distance between cursor and target center. 

In the pursuit task with point-to-point trajectories (see Figure 1D, bot- 
tom), the target moved smoothly from one randomly generated waypoint 
to another. The waypoint positions were drawn from a uniform distri- 
bution on the 76.8 cm wide by 57.6 cm tall (in screen scale) work space 
and were not visible to the monkeys. We limited the target’s acceleration 
(<3.2 cm/sec, screen scale) and velocity (<9.6 cm/sec, screen scale) to en- 
sure smooth trajectories. Rewards were larger when the cursor was closer 
to the center of the target to motivate the monkey to pursue the target 
accurately. 


2.3 Data Preprocessing. To estimate the instantaneous firing rate, we 
counted spikes in 100 ms wide, nonoverlapping (i.e., spaced at 100 ms) 
time bins. The joystick position was sampled at 1 KHz and downsampled 
to 10 Hz. Velocity was calculated from position by two-point digital dif- 
ferentiation after the downsampling. We divided position, velocity, and 
binned spike counts by their standard deviations. We subtracted the means 
from position, velocity, and binned spike counts during closed-loop BMI 
experiments; in offline reconstructions, instead of centering kinematics and 
neuronal data, we added a bias term to the neuronal tuning (observa- 
tion) model. During experiments, the monkeys’ heads were not restrained, 
and they were allowed to take breaks from task participation. To exclude 
data recorded while the monkeys did not participate, we used two meth- 
ods. First, in both closed-loop and offline experiments, we excluded data 
recorded while the monkey did not hold and move the joystick. If the joy- 
stick did not move for more than 2 or 3 seconds, depending on the session, 
the monkey was considered not to be participating. During sessions of BMI 
control, the disconnected joystick was still held and moved by the monkey 
(brain control with hand movements). Second, during closed-loop experi- 
ments, we found that the monkey would sometimes hold the joystick but 
look away from the screen and rest. We thus excluded data recorded while 
the monkey looked away from the screen from update calculations. The 
experimenter watched the monkey and toggled a button in the experiment 
software when the monkey looked away. 

We conducted offline analyses using Matlab software (MathWorks). For 
closed-loop experiments, we implemented the proposed adaptive decoding 
method in C++ as part of a custom-built BMI software suite running on a 
desktop computer with an Intel Core i7 processor. 


2.4 Evaluation Procedures. We quantified accuracy of reconstructions 
and control using the signal-to-noise ratio (SNR). SNR is commonly used in 
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engineering and was used in previous studies on BMIs (Sanchez et al., 2002; 
Kim et al., 2003). SNR is the inverse of the normalized mean squared error, 
with larger numbers indicating more accurate control. SNR is typically 
reported in decibels (dB), which transforms the ratio with a logarithm to 
allow arithmetic averaging. The formula for the SNR is 


(2.2) 


VAR 
SNR,p = 10 x logy, 


MSE 


where VAR is the variance of the desired values and MSE is the mean 
squared error of the predicted values from the desired values. For offline 
reconstructions, the desired values are the hand trajectories recorded by 
the joystick, and the predicted values are the outputs of the decoder. For 
closed-loop control, desired values are the target trajectories of the pursuit 
task, and the predicted values are the output of the decoder. In this study, 
we analyzed the SNR of the position values of the trajectories. We analyzed 
x- and y-coordinates separately and combined them with arithmetic aver- 
aging. When we calculated SNR in 1 minute windows of a session, we used 
the same signal variance (numerator), the value from the entire session, 
for every window, to allow comparison among windows. When compar- 
ing accuracy among conditions, we computed the SNR in decibels for each 
condition and then subtracted the values. 

To ease comparison to previous work, the following equation provides 
the approximate correlation coefficient (Pearson’s r) corresponding to an 
SNR value in dB, under two assumptions: the mean of both the desired and 
predicted signals is 0, and the variances of the desired and predicted signals 
are equal. Without these assumptions, correlation coefficient and SNR are 
generally incomparable. The equation is 


1 


ry 1 To Speen 
2. 106isSNRp) 


(2.3) 


For the derivation and details on the approximations taken, see the 
appendix. 


3 Computational Methods 


3.1 Overview. Bayesian regression self-training updates the model 
of neuronal tuning to movements used by the decoder by periodically 
performing Bayesian regression on the decoder output and neuronal activ- 
ity (see Figure 1C). This paradigm uses the prediction algorithm’s output 
to update its parameters, hence, we call it self-training. Self-training is fea- 
sible for updating neuronal tuning models because of the redundancy in 
motor representation in large populations of neurons (Narayanan, Kimchi, 
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& Laubach, 2005). For example, when a neuron changes its tuning, the de- 
coder can still use the tuning of the rest of the population to predict desired 
movements. The movement information conveyed by the changed neu- 
ron’s modulation is drowned out by the movement information conveyed 
by the entire population. A small error due to the outdated tuning model for 
the changed neuron will contaminate predictions, since the Kalman filter 
combines information from all the neurons in a weighted average. How- 
ever, if the change in tuning is small or the number of neurons that change 
simultaneously is small compared to the size of the population, the error 
in predictions using the entire population will be small, allowing accurate 
enough predictions with which Bayesian linear regression can update the 
tuning model parameters of the changed neurons. 

We use a fifth-order unscented Kalman filter (Julier, Uhlmann, & 
Durrant-Whyte, 1995) with a quadratic model of neuronal tuning for the 
decoder (see Li et al., 2009 for implementational details). This decoder is 
an extension of the well-known Kalman filter (Kalman, 1960) Bayesian esti- 
mation algorithm, which has been used for BMI decoding (Wu et al., 2003, 
2006; Gilja et al., 2009, 2010). It differs from previous Kalman filter decoders 
by incorporating a quadratic model of neuronal tuning (Moran & Schwartz, 
1999; Li et al., 2009) and the unscented transform (Julier et al., 1995) approxi- 
mation mechanism needed to compute posteriors with the nonlinear tuning 
model. It also allows the tuning model to capture tuning across multiple 
time offsets by keeping multiple time taps of kinematic variables in its state 
(five time taps in this study). 

Between tuning model updates, we store the output of the unscented 
Kalman filter (UKF) decoder and the neuronal activity. Before each up- 
date, we preprocess the decoder output by performing Kalman smoothing 
(Rauch et al., 1965). This process improves the accuracy of the estimates 
by propagating information from estimates of later states to estimates of 
earlier states using the movement model. For example, if the user is trying 
to move a BMI-controlled cursor to a particular location, the target location 
(decoded later in time) can improve the estimate of the movement trajec- 
tory. This implements a probabilistic mechanism for improving the data 
used for updates. This approach is similar in spirit to the heuristic methods 
found to be successful in prior work: using a point in between the BMI 
cursor position and target as the training signal (Shpigelman et al., 2009) 
and rotating the decoded velocity vector toward the target in the training 
signal (Gilja et al., 2010). 

The smoothed trajectories (independent variable) and the concurrent 
neuronal activity (dependent variable) are the inputs to the Bayesian linear 
regression. The Bayesian linear regression uses the previous tuning model 
parameters as priors and computes the posteriors on the parameters given 
the stored data. The updates occur in the background (e.g., in a low-priority 
thread) while the UKF continues to decode in real time using the existing 
tuning model. Once the update completes, the UKF uses the new tuning 
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model and storing of outputs for the next update begins. To compute the 
update, we use two formulations of Bayesian linear regression. One variant 
uses a joint distribution for the tuning model coefficients and allows analyt- 
ical calculation of the solution. The second variant is more flexible, allowing 
temporary omission of some neurons from updates and addition of newly 
discovered neurons, but it requires a factorized distribution for the tuning 
model coefficients and an approximate solution using variational Bayes. 


3.2 Tuning Model. We used a fifth-order UKF with two future taps and 
three past taps, which means each spike may relate to the desired movement 
occurring within a time difference of 300 ms (Li et al., 2009). This setting 
is consistent with neurophysiological data and provides accurate decoding 
without requiring too many parameters. The fifth-order UKF used only 
one tap of kinematics in its movement model to minimize the decoding 
contribution from exploiting patterns in the movement. The tuning model 
described the normalized firing rate of a neuron as a linear combination 
of the position, distance from the center of the work space, velocity, and 
the magnitude of velocity (speed). The magnitude terms have been found 
to model neuronal activity well (Moran & Schwartz, 1999) and improve 
decoder accuracy (Li et al., 2009). The tuning model is 


* 
hia. XPOS; _3. 
0S 
hing YP0S;_34% 
5 2 2 
Nis P05? 44 TYPOS; 34% 
it = 3s h xvel @2) 
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Here, z;, is the predicted binned spike count of neuron i at time t, xpos, 
and ypos, are the x and y coordinates of the cursor at time t, and xvel, and 
yvel, are the x and y velocities of the cursor at time ¢. The distance and 
speed terms make the tuning model quadratic in position and velocity and 
require the use of an unscented Kalman filter for prediction. h ,,.--, Nig 5 
are 30 scalar parameters for neuron i. We use the N x 30 matrix H to hold 
the parameters for all neurons in the population, where N is the number of 
neurons. The k variable iterates over the time taps in the state variables (see 
Li et al., 2009, for details of the nth-order model). 
The model for the noise in the quadratic tuning model is 


Vit = Zit + Vit» (3.2) 
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where y;, is the actual binned spike count for neuron i at time t, and v;, 
are elements of a noise vector v,, which is drawn independently from a 
multivariate normal distribution with zero mean and covariance R. R is 
an N x N matrix. The matrices H and R constitute the tuning model or 
observation model used by the UKF. To facilitate Bayesian updates of these 
two matrices, we store them as probability distributions and operate on the 
distribution parameters. 


3.3 Bayesian Regression. The model for the multivariate linear regres- 
sion between the vector of population neuronal activity y and the vector of 
kinematic features x is 


y, = Hx, +€ e ~N(0,R). (3.3) 


To perform a Bayesian update of the parameters, we need a formulation 
for the prior and posterior distributions of H and R. It is often desirable 
for the prior and posterior distributions to be from the same distribution 
family and differ only in their hyperparameters (conjugacy). Conjugacy is 
advantageous because it may allow analytical computation of the hyperpa- 
rameters of the posterior (Bernardo & Smith, 2000). After an update, we use 
the mean of the posterior distribution as the tuning model for subsequent 
decoding. 

We examined two formulations for the distribution on H and R, which 
allow conjugacy. The first formulation represents the prior and posterior 
joint probability of H and R in a normal-inverse-Wishart distribution. This 
formulation allows exact, analytical calculation of the Bayesian regression 
posteriors, and the formulas for computing the posteriors are well-known 
(Bishop, 2006; Rencher & Schaalje, 2008). However, it does not allow updates 
on subsets of neurons or the addition of newly discovered neurons. Thus, we 
explored an alternative formulation. The alternative formulation represents 
the tuning coefficients of each neuron as a separate multivariate normal 
distribution. Analytical computation of the regression on this formulation 
is not possible, and we use a variational Bayesian solution. This method 
is a multiple-predicted-variable extension of the single-predicted-variable 
Bayesian linear regression model presented by Bishop (2006). 


3.4 Joint Formulation. The probability model for Bayesian regression 
with a joint distribution for H and R is shown in Figure 2A and specified 
below: 

y,|x,, H, R ~ NV (Hx,,R), (3.4a) 
{vec (H), R} ~ NW"! (vec (uw), AW, Wm). (3.4b) 
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Figure 2: Graphical models for (A) joint distribution Bayesian regression, and 
(B) factorized distribution variational Bayesian regression. Ellipses indicate ran- 
dom variables, double ellipses indicate observed random variables, rectangles 
indicate observed point values, and rectangles indicate repeated nodes. 


x, is a column vector of length D of regression features at time t, where D 
is the number of kinematic features (30 for the experiments in this study). 
Features are calculated from the mean of the smoothed outputs. y, is a 
column vector of length N of binned spike counts at time t. The vec() 
function converts a matrix into a column vector by vertically concatenating 
the columns, with the left-most column at the top. The variables jz (N x D), 
A (D x D), @ (N x N), and m (scalar) are hyperparameters that describe the 
normal-inverse-Wishart distribution for the tuning model parameters: 


NW! (vec (H), Rivec (wu), A7!, , m) 
= N (vec (H) |vec (1), A @ R) W! (RIW, m). (3.5) 
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Here, ® is the Kronecker product. Due to conjugacy, the posterior parame- 
ters are the prior parameters updated by statistics of the neuronal activity 
and kinematic features. When algebraic manipulations on the likelihood 
and the prior are performed, the following update equations can be found: 


A<—A+XX', (3.6a) 
uw <— (fA + YX")AT, (3.6b) 
WW 4+ fA’ +YY'—wAn', (3.6c) 
m<—m+T. (3.6d) 


Tildes indicate prior values for the hyperparameters. X is the D x T matrix 
of features over the T time points in the update batch. Y is the N x T 
matrix of binned spike counts. The prior hyperparameters ji, A, VW, and 71 
are the posterior hyperparameters from the previous update or the initial 
parameter fit (see section 3.8). 

Despite the advantages of offering a fast and exact analytical solution, 
this formulation has the disadvantage of linking the confidence measure 
on coefficients from different neurons through the A hyperparameter. This 
means that we cannot update the coefficients of neurons separately. The 
ability to update subsets of neurons is important because this allows us to 
halt updating for a portion of neurons, which is useful if some neurons are 
affected by transient recording noise or are stable in tuning. 

Furthermore, adding newly discovered neurons to the decoder in a prin- 
cipled manner means using low initial confidence for their coefficients. 
Adjusting the A hyperparameter for new neurons is not possible since 
the entire population shares this hyperparameter. While changing the new 
neurons’ entries in W affects their initial confidence, this approach does not 
increase the magnitude of future updates. That is, during subsequent up- 
dates, we expect the magnitude of updates for new neurons to be larger, 
if all other factors are equal, since the confidence about the new neurons’ 
coefficients should be lower than the confidence about the coefficients of 
neurons already in the model. Changing W means that in subsequent up- 
dates, all neurons are treated as if the confidence in their coefficients is 
the same (see equations 3.6a and b). Alternatively, one may use separate 
A terms for each neuron. Unfortunately, using separate A terms for each 
neuron breaks conjugacy and precludes analytical computation. In the next 
section, we present an approximate solution for the case where each neuron 
has a separate A hyperparameter. 


3.5 Factorized Formulation. To allow updates on subsets of neurons, we 
factorized the distribution for the tuning coefficients so that the coefficients 
for each neuron are represented by a multivariate normal distribution. Since 
this formulation is no longer conjugate to the likelihood of linear regression, 
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we applied variational Bayes (Jordan, Ghahramani, Jaakkola, & Saul, 1999), 
an approximate inference technique for hierarchical probability models. 
Other studies have applied variational Bayes to decoding neuronal activity 
(Ting et al., 2005, 2008; Chen, Kloosterman, Wilson, & Brown, 2010). 

The variational Bayesian approach assumes a factorization of the joint 
posterior of the hidden random variables (tuning parameters) given the 
visible random variables (neuronal activity and smoothed decoder out- 
puts). The factorization facilitates conjugacy in the links of the resulting 
probability model and allows the use of an iterative, fixed-point procedure 
to find the approximate posterior on the hidden variables. The fixed-point 
procedure can be thought of as a Bayesian generalization of the expectation- 
maximization (Dempster, Laird, & Rubin, 1977) algorithm. The procedure 
minimizes the Kullback-Leibler divergence between the true joint poste- 
rior on the hidden variables and the factorized posterior. The steps in the 
procedure do not increase the divergence, which guarantees convergence, 
but potentially to a local optimum. We refer readers to works by Jordan 
et al. (1999), Winn and Bishop (2005), and Bishop (2006) for more details on 
variational Bayes. 

We will call the variational Bayesian solution to the factorized Bayesian 
regression problem variational Bayesian regression (VBR). To facilitate con- 
jugacy, we separated the distributions for the tuning coefficients H and the 
tuning noise covariance matrix R. The probability model for VBR is shown 
in Figure 2B and specified as 


y,|x,, H, R~ NV (Hx,, R), (3.7a) 
h) ~N(u;, Aj’), (3.7b) 
R~w wm). (3.7c) 


Here, i is the index on neurons. h; is row i of H. jz; are the mean hyperpa- 
rameters for the tuning model coefficients of each neuron (column vectors 
of length D). A; are the precision hyperparameters of the tuning coefficients 
for each neuron (each D x D). W (N x N) and m (scalar) are hyperparameters 
for the tuning noise covariance R. Despite the factorization among neurons 
of the tuning coefficient distributions, activity among neurons may still be 
correlated due to other factors. Thus, we use the inverse-Wishart distribu- 
tion to represent the distribution on a full noise covariance matrix. 


3.6 Variational Approximation. In the following, we denote the inferred 
posterior distributions for h, and R with the Q-functions Q(h;) and Q(R), 
respectively. Furthermore, we denote the product of Q(h;) as Q(H). 

Variational Bayes minimizes the Kullback-Leibler divergence between 
the true posterior and calculated posterior by maximizing a related lower 
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bound £(Q) (Winn & Bishop, 2005): 
In P(X, Y) = L(Q) + KL(Q||P). (3.8) 


Here, Qis shorthand for Q(H)Q(R), and P is shorthand for the true posterior 
P(H, R|X, Y) and 


P(H,R, X, Y) 

L = H R) ln ——_—- , 3.9. 

@=| Qanowmn Te (3.9a) 
P(H, R|X, Y 

KL(Q\|P) =— i. QR) In (3.9b) 


L£(Q) is the lower bound on the log marginal probability In P(X, Y) as the 
Kullback-Leibler divergence is nonnegative. Since In P(X, Y) is fixed given 
the filter outputs and neuronal activity, by maximizing £(Q), we minimize 
the Kullback-Leibler divergence. Winn and Bishop (2005) show that £(Q) 
is maximized when 


1 
Qth;) = > exp(ln PCH, R, X, ¥)) om): (3.10a) 


1 
Q(R) = = exp(In P(A, R. X, Y)) oR): (3.10b) 
R 


where the Z terms are the normalization factors needed to make the Q- 
functions valid probability distributions. The angled brackets () indicate 
expectations, which are taken with respect to the other Q-functions, as 
indicated by the subscripts. 

The parameters of Q(h;) and Q(R), which maximize the lower bound are 
found using a set of fixed-point equations derived from the above conditions 
by factorizing the graphical model and performing algebraic manipulations 
(Winn & Bishop, 2005). These equations update the hyperparameters of 
each Q-function. The updates are executed in turn for each Q-function until 
convergence is detected by a lack of change in £(Q). 

The prior hyperparameters ji;, A;,W, and 7 are the values from the 
previous VBR update or the initial parameter fit. At the beginning of the 
fixed-point iterations, we initialize j4,, A;, ¥, and m to their prior values. 

The update for H is 


A; A,;+XX™(R1)G) i=1,....N, (3.11a) 


BoA (Afi tXy RY GQ) i= LN, (3.11b) 
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Tildes indicate prior values of hyperparameters. y; is the length T row 
vector of binned spike counts for neuron i. (R~') ) is the (i, 7) entry of the 
expectation of R7!: 


Gi 


RY G9 = (mo), 5: (3.12) 
The update for R is 

m<—m+T, (3.13a) 

W <— W + (Y— (H)X)(Y — (H)X)' + W. (3.13b) 


(H) is the matrix formed by the vertical stacking of the row vectors j). W 
(N x N) is a temporary matrix used to account for the covariance of tuning 
coefficients: 


tr(X'A>!X), ifi=j 
“=| alae J . (3.14) 


0, otherwise 


Wij is the (i, 7) entry of W, and tr() denotes the trace operator. 

Once the posterior values of ;, A;, YW, and m are found, the posterior 
expectations for H and R can be computed for use in subsequent decoding. 
(H) is described above. (R) is 


Vv 


aN 


(3.15) 


3.7 Lower-Bound Computation. We calculate the lower-bound L(Q) 
to detect convergence. It also serves as a useful diagnostic, since it should 
not decrease after an update on any random variable. The equation for the 
lower bound can be rewritten as 


£(Q)= / Sogn 
~ Sr Q(H)Q(R) 
= (In P(H, R, X, Y)) — (InQ(H) + InQ(R)). (3.16) 


The expectations are taken with respect to Q(H) and Q(R). By factorizing 
the graphical model (Winn & Bishop, 2005), we decompose the lower bound 
as the sum of three terms £;,, Lp, and L,: 


Ly = (In P(A iL, wn) — (InQ(A))p, (3.17a) 
Lp= (InP(R|Y, tit), — (InQ(R))g, (3.17b) 


L, = (In P(Y|X, H, R))iR- (3.17c) 
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After substituting the probability density functions, rearranging, and 
taking expectations, we use the following expressions for £;;, Lp, and L, to 


calculate the lower bound: 


ie 
ij ivi M; 
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where 
No (m+1-i 
(In|R7'|) =NIn2+ In|W4| + Yop (“) 
i=1 
(In|R]) =—(n|R™}), 
((Hx,) "R7 (Hx, )) = ((H)x,) (R™) (A) x)" + tr, (R™)), 
oe Ua Fi ifi=j 
Bay| ‘ a 
0, otherwise, 


(Ro) =m, 


(3.18) 


(3.19) 


(3.20) 


(6.21) 


(3.22) 
(3.23) 


(3.24) 


(3.25) 


Z, isan N x N temporary matrix. I'() is the gamma function, and y() is the 


digamma function. 
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3.8 Initial Fit. We fit the initial neuronal tuning model using Bayesian 
regression so that tuning model parameters were described by distributions 
instead of point estimates. For parameters updated with joint formulation 
Bayesian regression, we used the joint formulation Bayesian regression for 
initial fitting; for parameters updated with factorized formulation Bayesian 
regression, we used the factorized formulation Bayesian regression for ini- 
tial fitting. 

During initial fitting of tuning model parameters, we used priors that 
have a similar effect as the regularization in ridge regression: 


ji; =0, (3.26a) 
A, =)71, (3.26b) 
W =I, (3.26c) 
m=N +2. (3.26d) 


0 is the zero matrix and I is the D by D identity matrix. For joint formulation 
Bayesian regression, we have only one ji and one A. For factorized formula- 
tion Bayesian regression, one ji; and A; for each neuron must be initialized. 
The prior variance i of the coefficients acts in the same way as the ridge 
regression regularization parameter. In reconstructions, we chose the value 
for 4 by optimizing for decoder accuracy. For closed-loop experiments, we 
used the optimal from reconstructions. When fitting the movement model 
for the unscented Kalman filter, we used 4 = 10~° and m = 6. 

We fitted the initial tuning model using hand movements and neuronal 
data collected during joystick control of the task cursor. We did not address 
the problem of fitting models in the absence of a training signal, as numerous 
studies (Taylor et al., 2002; Wahnoun, Helms Tillery, & He, 2004; Wahnoun, 
He, & Helms Tillery, 2006; Hochberg et al., 2006; Donoghue, Nurmikko, 
Black, & Hochberg, 2007; Velliste, Perel, Spalding, Whitford, & Schwartz, 
2008) have addressed this problem. Instead, we focused on the problem 
of adaptive decoding (after the initial fitting) without a training signal, 
since in a clinical setting, a dedicated training procedure for initial fitting is 
reasonable, but frequent user retraining for maintaining decoder accuracy 
is burdensome. 

The movement model of the UKF was also fitted using hand movement 
training data and Bayesian regression using the procedure described above. 
However, the movement model was not updated. At the start of decoding, 
we set the UKF’s initial state to the mean state in the training data and the 
initial covariance to the state covariance in the training data. 
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3.9 Adding and Omitting Neurons. For the factorized formulation vari- 
ational Bayesian regression, neurons may be added or omitted by adjusting 
Lu; A;, and W. 

One may add a new neuron i by adding a new ju; vector and a new 
matrix A,, and by increasing each dimension of W by 1. The values of the 
new entries encode prior knowledge of the new neuron’s tuning, similar to 
the prior values used for the initial fit described above. An uninformative 
prior could be encoded with 


,=9, (3.27a) 
ASHI, (3.27b) 


where A is a small positive number. 

For the new entries in YW, one may use m x a for the new diagonal (i, i) 
entry and Os for the new nondiagonal entries in the ith row and ith column. 
This encodes a strong prior for the value a in the diagonal entry of R for the 
new neuron. To encode a noninformative prior, one could set the (R7') ;; 
terms to 1 in equation 3.11 in the first VBR iteration (before W is updated), 
set the ith row and column of to 0 in equation 3.13, and multiply the ith 
row and column of W by m/T after using equation 3.13. 

One may omit a neuron i from an update (or permanently remove it 
from decoding) by removing its 4; vector, A; matrix, and the ith row and ith 
column from W. For neurons that are temporarily omitted, one may wish 
to adjust the ith row and column of W so that the omitted neuron’s entries 
in R remain the same, since m changes for the entire population during an 
update. This can be done by multiplying by m.,,,,,/m, where m.,,,,,, is the new, 
larger value of m after the update. 

Multiple neurons may be added or omitted by repeating the above pro- 
cedures. 

In closed-loop experiments, we omitted neurons from decoding and up- 
dating if they were flagged inactive, based on two tests. First, if a neuron 
produced no spikes during the self-training data window, it was flagged 
inactive. Second, a likelihood ratio test was employed during filtering in 
some sessions. In the likelihood-ratio test, the product over the previous 
10 seconds of the likelihood ratio between two models was calculated and 
compared to two thresholds. In the numerator of the ratio is the likelihood 
of the active model. In this model, the filter innovations for neuron i follow 
a normal distribution with zero mean and variance equal to its diagonal 
entry in the innovation covariance matrix. In the denominator is the like- 
lihood of the inactive model, in which the observations (not innovations) 
for neuron i follow a normal distribution with zero mean and the same 
variance as the active model. If the product of the likelihood ratio over the 
previous 10 seconds fell below 10-1", the neuron was flagged inactive. If 
the product of the likelihood ratio rose above 1, the neuron was flagged 
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active. The thresholds were set by hand for a low false-positive detection 
rate for inactivity. 


3.10 Baseline Firing Rates. The quadratic model of tuning described 
previously operated on binned spike counts with their means, or baseline 
firing rates, subtracted. Studies have shown that baseline firing rates of neu- 
rons change over time (Chestek et al., 2007; Ganguly & Carmena, 2009) and 
so the means should be updated. We explored two methods for updating 
the means in offline reconstructions: (1) using a time-varying bias term in 
the quadratic model of tuning without the mean subtraction and (2) using 
the sample means of the binned spike counts in the update window as the 
new baseline firing rates, updated once per update window. We found little 
difference in these two approaches. In the results, we report reconstructions 
made with the bias term approach and closed-loop experiments performed 
with the sample mean approach. 

To check if only updating the baseline firing rates for each neuron was 
sufficient to retain control accuracy, we updated the baseline firing rates 
of the static UKF (control condition) in the last two closed-loop sessions. 
To do this, we computed exponential moving averages on the binned spike 
counts. The half-lives of the moving averages were between 1 and 3 minutes. 


3.11 Transition Model for Tuning Parameters. Since we assume the 
tuning model parameters vary over time, we need a model of how they vary 
in order to track them. This model is a transition model on the tuning model 
parameters. The update rules of the transition model described below are 
executed before the Bayesian linear regression of each self-training update. 

For the tuning model coefficients H, we used the identity transition 
model with normally distributed noise. We called the amount of noise the 
model drift parameter 5. This transition model increases the covariance of 
the coefficients by the model drift: 


A> 2 A OL (3.28) 


lis the D x D identity matrix. For joint formulation Bayesian regression, 
the single A is updated; for factorized formulation Bayesian regression, the 
A, for each neuron is updated. 

For the tuning model noise covariance matrix R, we increased the vari- 
ance of the inverse Wishart distribution without changing the mean by 
scaling m and W by the same value (less than 1). We applied the scaling 
when m exceeded a degrees of freedom cap 1: 


Wo _W, (3.29a) 


3 


m <— mM. (3.29b) 
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We applied the scaling when m exceeded the cap because this approach 
allowed the use of more data to estimate R initially and produced slightly 
more accurate reconstructions. With the correct settings, the two approaches 
can behave in the same manner asymptotically. However, the scale-at-every- 
update approach can handle updates at different frequencies by adjusting 
the scaling factor. 


3.12 Kalman Smoothing. The Kalman smoother, also known as the 
Rauch-Ting-Striebel smoother (Rauch et al., 1965), combines the Kalman 
filter (forward pass) and a backward smoothing pass to form the forward- 
backward algorithm for linear-gaussian systems. The backward pass of the 
Kalman smoother starts from the most recent filter output and iterates to 
the earliest filter output in the update window: 


& = FR PKR, (3.30) 
where 

F, = F'‘d—- Qe@P,F' +Q)*°), (3.31) 

K, = F''Q(FP,F'+Q)"". (3.32) 


%, is the smoothed state at time t, P, is the state covariance at time t, F is the 
movement model matrix, and Q is the movement model noise covariance 
matrix. These equations describe one iteration of the backward pass of the 
smoother. To initialize the iterations, we set X, to x;, the most recent filter 
output. 

Note that since we use a linear transition (movement) model, the stan- 
dard Kalman smoother is applicable. 


4 Results 


4.1 Offline Reconstructions. We first performed offline reconstructions 
using 18 sessions in which monkeys controlled the cursor using a joystick. 
We discarded the first minute of each session and used the subsequent 2 
minutes of hand movement (measured by the joystick) and neuronal data 
to fit the initial tuning and movement models. We used Bayesian regres- 
sion with priors as described in section 3.8 to perform the initial fits. Then 
the fifth-order unscented Kalman filter (UKF) with and without Bayesian 
regression self-training updates (occurring every 2 minutes) reconstructed 
the hand movements for the remainder of the session from neuronal data 
only. We performed reconstructions with both variants of Bayesian regres- 
sion and repeated for different settings of the model drift parameter (see 
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section 3.11). Although our update method did not require a priori knowl- 
edge of the desired movements of the BMI user, we speculated that if we 
used the recorded hand movements (representing desired movements) for 
updates, accuracy of the updates would improve due to reduced noise in 
the training data. We also wanted to compare our update method to previ- 
ous work that used hand movements for updates. Thus, we also performed 
reconstructions using hand movements as input to the Bayesian regression 
updates. 

To summarize, the experimental conditions were nonupdated UKF fit- 
ted with joint distribution Bayesian regression (static-BR-fit), nonupdated 
UKF fitted with factorized distribution variational Bayesian regression 
(static-VBR-fit), UKF updated with joint distribution Bayesian regression 
self-training (BR), UKF updated with factorized distribution variational 
Bayesian regression self-training (VBR), UKF updated with joint distribu- 
tion Bayesian regression on hand movements (BR-hand), and UKF updated 
with factorized distribution variational Bayesian regression on hand move- 
ments (VBR-hand). There are two static conditions (static-BR-fit and static- 
VBR-fit) because the fits calculated with Bayesian regression and variational 
Bayesian regression are slightly different. 

Figure 3A shows the position reconstruction accuracy of five conditions 
(we omitted static-VBR-fit for clarity) quantified using a signal-to-noise 
ratio (SNR, in decibels; see section 2.4) and grouped by monkey. Table 1 
shows the results for each session. We tried multiple settings of the model 
drift parameter; Figure 3A and Table 1 show the accuracy for the empirically 
best parameter setting found on the test data. 

BR was significantly more accurate than static-BR-fit (two-sided, paired 
sign test, n = 18, a = 0.05, p < 0.001) with a mean difference of 0.54 dB. 
VBR was significantly more accurate than static-VBR-fit (p < 0.001) with a 
mean difference of 0.50 dB. BR was significantly more accurate than VBR 
(p < 0.05) with a mean difference of 0.10 dB. Static-BR-fit was significantly 
more accurate than static-VBR-fit (p < 0.05) with a mean difference of 0.06 
dB. BR-hand was significantly more accurate than BR (p < 0.05, 0.50 dB 
mean difference) and static-BR-fit (p < 0.002, 1.04 dB mean difference). 
VBR-hand was significantly more accurate than VBR (p < 0.05, 0.57 dB 
mean difference) and static-VBR-fit (p < 0.002, 1.07 dB mean difference). 

Figure 3B shows the accuracy improvement of BR versus static-BR-fit 
and VBR versus static-VBR-fit for different settings of the model drift pa- 
rameter. We chose the model drift parameter settings to be equally spaced 
in logarithmic space. Different sessions had different optimal values of the 
model drift. We did not optimize the model drift parameter on the training 
data, as 2 minutes of training data was too brief to have dramatic changes 
in tuning. Since the best model drift parameter was found using the test 
data, the best accuracy is an optimistic estimate. However, using the best 
overall value (based on mean across sessions) of e~!° ~ 4.5. 1075 results in 
significant improvement versus static (BR: mean 0.36 dB, p < 0.01; VBR: 
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Figure 3: Position reconstruction accuracy of UKF with and without updates. 
(A) Bars summarize mean accuracy for each monkey and condition. The static 
condition used parameters fit with joint distribution Bayesian regression. BR- 
hand and VBR-hand conditions used hand movements for updates. Accuracy 
values are from the best setting of the model drift parameter found using the 
test data. (B) Accuracy improvement versus model drift parameter. Sessions are 
indicated by curve labels. 
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mean 0.33 dB, p < 0.01). Accuracy as a function of model drift appears 
unimodal. 

We examined the time course of accuracy differences between up- 
dated and static decoders in each session by comparing the improve- 
ment (BR minus static-BR-fit, VBR minus static-VBR-fit), calculated in 1 
minute windows, versus the time in the session. The mean (across sessions) 
slope of the best-fit linear trend line of the improvement versus time was 
0.036 dB/min for BR, 0.033 dB/min for VBR, 0.078 dB/min for BR-hand, 
and 0.082 dB/min for VBR-hand. 

We examined the benefit of using the Kalman smoother to smooth 
decoder outputs before use in self-training. We performed offline 
reconstructions without using the Kalman smoother and compared the 
reconstruction accuracy. For BR, Kalman smoothing improved accuracy by 
an average of 0.62 dB. For VBR, Kalman smoothing improved accuracy 
by an average of 0.80 dB. Both improvements were statistically significant 
(p < 0.002). The BR and VBR adaptive decoders without Kalman smooth- 
ing were nominally less accurate than static decoding (by 0.082 dB and 
0.30 dB, respectively), but the difference was not statistically significant 
(p > 0.05). 

The Bayesian regression self-training updates were computationally fast. 
The slowest step during each update was the Kalman smoothing backward 
pass. With a Matlab implementation on a Intel Core i7 class desktop com- 
puter, each BR update in the reconstructions, including preprocessing, used 
1.57 + 0.002 seconds (mean + standard deviation). Each VBR update used 
1.97 + 0.13 seconds. While these execution times are longer than the 100 ms 
binning interval, updates can execute in a low-priority thread in a real-time 
BMI system, the approach taken in our closed-loop experiments. 


4.2 Closed-Loop BMI. We conducted closed-loop BMI control experi- 
ments with monkey M in 11 sessions over 29 days. We used variational 
Bayesian regression self-training updates to handle transient recording 
problems better. When a neuron had no spikes during an update win- 
dow or if a likelihood ratio test flagged it as temporarily quiet, it was not 
included in updates and decoding. 

Monkey M performed the point-to-point variant of the pursuit task in 
each session. We set spike-sorting templates by hand using Plexon spike- 
sorting software at the beginning of the first session and kept the templates 
fixed for the remaining sessions. We made no attempt to select neurons with 
stable waveforms; we used all 139 sorted units. 

In the first session, we used 2 minutes of hand-controlled behavior to 
fit the initial tuning and movement model parameters using variational 
Bayesian regression. In each session, the monkey controlled the cursor us- 
ing the fifth-order UKF and two sets of parameters in turn. The first set (static 
condition) consisted of the parameters fit from the hand-controlled behav- 
ior. These parameters were not updated and were the same for each session. 
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The second set (VBR condition) was updated by variational Bayesian 
regression self-training through the course of the experiments. The sec- 
ond set was initially equal to the parameters fit from the hand-controlled 
behavior, but updated repeatedly with changes saved between sessions. In 
other words, we stored the VBR condition parameters at the end of each 
session and used them in the subsequent session, where they were updated 
further. In each session, monkey M first performed the pursuit task with 
the static parameters for 6 to 10 minutes. Then the monkey performed the 
pursuit task with the updated parameters for 10 to 50 minutes. The variance 
in time was due to the monkey’s participation. An update occurred each 
time 2 minutes of self-training data were collected; since we did not use de- 
coder predictions made while the monkey was not looking at the screen and 
holding the joystick, updates occurred less frequently than every 2 minutes. 
The model drift parameter ranged between 10-3 and 10~°. We used larger 
values at the beginning of each session to compensate for the longer time 
duration since the previous update. 

Chestek et al. (2007) and Ganguly and Carmena (2009) found significant 
changes in baseline firing rates of neurons. We investigated whether updat- 
ing only the baseline firing rate parameter for each neuron could maintain 
control accuracy. We updated baseline firing rates with exponential moving 
averages of firing rates in the last two sessions for the static condition. 

To quantify control accuracy, we measured the SNR of the decoder output 
cursor position with respect to the target position. We calculated the SNR 
across the entire time segment of each condition and the best SNR among 
1-minute windows of each condition. We included the latter metric because 
it is less sensitive to practice and motivational effects. 

Figure 4A shows control accuracy for the 11 sessions over 29 days. The 
black curves indicate SNR over the entire condition (solid, static all) and 
best SNR among 1 minute windows (dotted, static best) for static. The 
gray curves indicate SNR over the entire condition (solid, VBR all) and 
best SNR among 1 minute windows (dotted, VBR best) for VBR. Note that 
no data were available for the static condition on day 6 because control 
was so poor that the monkey was not willing to perform the task for a 
period long enough to gather useful statistics. The UKF with VBR self- 
training updates maintained control accuracy much better than the static 
UKE. The slopes of the best-fit linear trendlines for each curve are: VBR all 
—0.088 dB/day, VBR best —0.070 dB/ day, static all —0.399 dB/day, and static 
best —0.487 dB/day. The update of the baseline firing rate parameters in the 
last two sessions for the static condition did not increase control accuracy 
to the level provided by the VBR self-training updated UKF, suggesting 
that updating the baseline firing rate parameters alone is insufficient for 
maintaining control accuracy. 

The solid curves in Figure 4B show control accuracy of VBR in 1 minute 
windows for each session. The highest values of these curves are the VBR 
best values in Figure 4A. The gray dotted lines indicate the SNR over the 
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Figure 4: Closed-loop BMI control accuracy with and without updates. 


(A) Solid curves 


indicate SNR over the entire condition time segment of each 


session, and dotted curves indicate the best SNR among 1 minute windows in 
the condition time segment of each session. In the last two sessions, we updated 
the baseline firing rate parameters of the static UKF using exponential moving 
averages. (B) Closed-loop BMI control accuracy versus time within each session. 
In four sessions, control accuracy was initially poor, but improved after two to 


three updates. 
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Closed-loop BMI control traces 
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Figure 5: Traces of closed-loop BMI control with the UKF updated with VBR 
and with a static UKF. Plots show x-axis position, black curves indicate the 
target location, and gray curves indicate the BMI controlled cursor location. 


entire VBR condition for each session (VBR all), and the black dashed lines 
indicate the SNR over the entire static condition for each session (static 
all). During days 6, 7, 17, and 28, control accuracy was initially poor but 
improved after two to three updates, demonstrating the ability of VBR self- 
training to adapt to changes in tuning or recording that had occurred since 
the previous session. The large variations in accuracy were due to the mon- 
key becoming unmotivated or frustrated and briefly stopping participation. 
When the monkey restarted participation, it had to reacquire the pursuit 
target; this process reduced the control accuracy metric. 

Figure 5 shows x-axis position traces of BMI control for the VBR and 
static conditions from sessions at the beginning, middle, and end of the 
closed-loop experiments. The traces show minutes 10 to 12 of control in the 
VBR condition and minutes 2 to 4 of control in the static condition. Black 
curves indicate the position of the pursuit task target, and gray curves in- 
dicate the BMI-controlled cursor. VBR self-training maintained control ac- 
curacy during the course of the experiments, while control with the nonup- 
dated UKF deteriorated. The oscillations seen in later sessions, especially 
for the static decoder, are about 0.5 to 2 Hz in frequency. We have observed 
similar oscillations in past experiments. The major departures from target 
pursuit are due to the monkey becoming frustrated and briefly stopping 
participation. 

We investigated the changes in the updated tuning model parameters 
of the UKF between the initial fit and the parameters on day 29. Figure 6A 


207 


Bayesian Adaptive Decoding for Brain-Machine Interfaces 3191 
A Absolute changes in coefficients B_ Changes in coefficient precisions 
4 Median 2.32x10° 
SN 0.3| Median 0.11 ‘7 patna 
o ! Mean 0.26 0.2). ii 
= it 
So 02 
2° 
> 
2 o.1 
® 
= 
s 
® oO 
Le x 10° 
Absolute change Change 
Cc Changes in baseline firing rates D Changes in tuning noise 
3 Median 3.1 Hz Median 1.8 Hz 
N 0.15) iMean 2.2 Hz 
o i 
E 0.2 
fe) 0.1 
= 
a 01 
2 0.05 
© 
=] 
3S 
i So 100 (SO 0~C*«S:S:*C«aSS 280 


Change, Hz 


Figure 6: Changes in tuning parameters detected by the adaptive decoder dur- 
ing closed-loop sessions. (A) Histogram of absolute changes in tuning model 
coefficients (H). (B) Histogram of changes in coefficient precisions (diagonals of 
A,). (C) Histogram of changes in baseline firing rates. (D) Histogram of changes 
in noise standard deviation (square root of diagonal of R). 


shows a histogram (normalized) of the absolute changes in tuning model 
coefficients (entries of H), with all coefficients from all neurons pooled 
into the sample. As we normalized the binned spike counts and kinematic 
variables to unit standard deviation before filtering, the units of the x-axis 
in the histogram are standard deviations of binned spike counts divided 
by standard deviations of kinematics. Dotted vertical lines indicate median 
and mean in the histogram. The large peak at 0 and the long tail indicate 
that a minority of coefficients changed while the majority did not. 

Figure 6B shows a histogram of the changes in tuning model coefficient 
precisions (diagonal entries of A;). The precisions quantify the certainty of 
the coefficient estimate and are equal to the inverse of the variances of the 
normal distributions for the coefficients. Larger precisions indicate more 
certainty. Units of the precisions are standard deviations of the kinematics 
squared divided by standard deviations of binned spike counts squared. 
Most coefficient precisions increased, indicating that certainty of coefficient 
estimates increased for most coefficients over the course of the experiments. 

Figure 6C shows a histogram of the changes in the baseline firing rate 
parameters for each neuron. Some neurons did not change baseline firing 
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rate, but there were also increases and decreases in the baseline firing rate. 
The mean and median of the changes were larger than 0 indicating a general 
increase in the firing rate. 

Figure 6D shows a histogram of the changes in tuning model noise stan- 
dard deviation entries (square root of diagonal entries of R). The tuning 
model of the UKF decoder includes both coefficients of tuning and a co- 
variance matrix of the noise in tuning, assuming that the noise follows a 
multivariate normal distribution. Figure 6D shows the changes in the square 
root of the diagonal entries of this covariance matrix. Larger standard de- 
viation entries mean more noise in the neuron’s firing due to modulation 
unrelated to the kinematic variables of interest, intrinsic randomness, and 
recording noise. The positive mean and median changes indicate that noise 
standard deviation entries increased overall. However, the standard devia- 
tion of some neurons decreased. At the end of the experiments, entries for 
three neurons were 0. 

In the initial fitting, 2 sorted units (of 139) were flagged inactive because 
they did not fire during the 2 minute training data. These two units received 
Os for all of their tuning coefficients. One of these units was later flagged 
active, and tuning parameters were fitted for it, demonstrating the ability of 
the adaptive decoder to add neurons. In total, 32 different units were flagged 
inactive for at least one update. The number of inactive units during each 
update fluctuated, and many inactive-flagged neurons became active again 
later. This was likely due to recording instability or noise. On days 1 through 
7,1 to 3 units were flagged inactive in each update. On days 13 to 17, 2 to 9 
units were flagged inactive in each update. On days 28 and 29, 9 to 16 units 
were flagged inactive in each update. The inactive neurons may explain 
some of the decrease in control accuracy for both the adaptive and static 
decoders. 


5 Discussion 


Bayesian regression self-training can improve reconstruction accuracy with- 
out having access to desired movements or using a priori assumptions about 
them. When knowledge of desired movements is available for update cal- 
culations, our method can produce even more accurate reconstructions (see 
Figure 3A). It is reasonable to suggest that the improvement in reconstruc- 
tion accuracy seen in sessions with length on the scale of dozens of minutes 
is partly due to the adaptive decoder improving the initial parameters fitted 
with the short 2 minutes of training data. This learning ability could exploit 
newly discovered neurons or neurons that become more tuned. Our method 
may also be adapting to changes in motivational or attentional state, as well 
as recording instabilities. 

The closed-loop experiments indicate that our adaptive decoder can 
maintain control accuracy better than a nonadaptive decoder (see Figure 4A) 
over almost a month. We believe the difference in control accuracy between 
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adaptive and nonadaptive decoders is partly due to tuning changes and 
partly due to recording instabilities. Though we fixed spike-sorting param- 
eters, we cannot be certain that the neurons recorded in each online session 
are the same. Also, electrode movements may change the shape and am- 
plitude of action potentials, leading to increases in spike sorting errors. 
These sorting errors may cause a portion of the spikes from a neuron to 
be dropped, which change the effective tuning properties of the recorded 
unit. The results indicate that the adaptive decoder can deal with changes 
in effective tuning due to recording instability (as compared to static). From 
this, we believe our adaptive decoder can handle real changes in neuronal 
tuning, even if they are not already present in our data. In the future, we 
hope to combine adaptive spike sorting (Wolf & Burdick, 2008; Gasthaus, 
Wood, Grr, & Teh, 2009) with adaptive decoding to address recording insta- 
bility directly. Alternatively, one may perform spike sorting in each session, 
find correspondences between neurons using metrics on neuron identity 
(Emondi, Rebrik, Kurgansky, & Miller, 2004; Wolf & Burdick, 2008), and 
add or delete neurons using the procedure described in section 3.9. 

While we cannot rule out the contribution of user practice effects to the 
differences between static and adaptive control accuracy, we note that the 
differences in accuracy are large even when comparing the best 1 minute 
window during static control to the overall adaptive control accuracy. The 
best 1 minute window metric should mitigate practice effects during the 
course of the static control condition. Furthermore, since the static and 
adaptive decoders had different tuning parameters, practice with one is 
not likely to help with the other. We also note that static decoding was 
conducted first, when the monkey was most thirsty and motivated, while 
adaptive decoding went on while the monkey became more and more 
satiated and less motivated. Near the end of the adaptive decoding con- 
dition, the monkey took breaks with increasing frequency and duration, 
until stopping altogether. These breaks reduce the SNR value, since the 
BMI cursor leaves the target and the target must be reacquired on re- 
sumption. Thus, any motivational effects should favor the static decoding 
condition. 

When Kalman smoothing was disabled in the offline reconstructions, 
adaptive decoding was not significantly better than static decoding. Since 
our update paradigm uses decoded trajectories as the training signal, the 
more accurate the decoded trajectories are, the better are the update results. 
We believe that the Kalman smoothing step in our reconstructions improved 
the accuracy of the decoded trajectory above a certain threshold. Above this 
accuracy threshold, the training signal was good enough so that the updated 
parameters could provide better subsequent decoding. We conjecture that 
if the decoder and neuronal recordings gave high enough accuracy before 
an update, then even without Kalman smoothing, the adaptive decoder 
would perform better than the static decoder since the training signal would 
already be good enough. In other words, we believe that although Kalman 
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smoothing helps improve adaptive decoding, it is not strictly necessary for 
adaptive decoding if baseline accuracy was high enough. 

The Bayesian regression updates in our decoder are computationally 
light and, due to their batch mode operation, can be performed in the 
background while decoding continues. By using a probabilistic model of 
how the tuning parameters change and by calculating Bayesian updates, 
our adaptive decoder essentially performs Bayesian filtering on the tuning 
parameters. However, the parameter filter operates at a lower frequency 
than the movement state filter. 


5.1 Changes in Neuronal Tuning. Studies have found that neuronal 
tuning to arm and hand movements may change over time. Taylor et al. 
(2002) observed changes in neuronal tuning as monkeys learned to operate 
a BMI based on a coadaptive algorithm. Li et al. (2001) and Padoa-Schioppa 
et al. (2004) reported that tuning characteristics of motor cortical neurons 
change as the result of adaptation to external force fields. In our previous 
experiments, we found that neuronal tuning to hand movements and to 
BMI-controlled cursor movements changed differently during prolonged 
BMI operation (Lebedev et al., 2005). Kim et al. (2006) used a bootstrap 
analysis to determine significance in tuning changes. They found significant 
changes in tuning that accumulated with time. Rokni et al. (2007) found 
tuning changes due to learning as well as additional slow, random changes. 
Chestek et al. (2007) found small but significant changes in tuning and 
baseline firing rates. However, they asserted that these changes were due to 
measurement noise, learning effects, and subtle behavioral changes rather 
than intrinsic randomness. Truccolo et al. (2008) examined tuning changes 
in neuronal data collected from two human subjects and found that 42% of 
neurons exhibited significant changes. 

The results from reconstruction and closed-loop experiments indicate a 
benefit of adaptive versus static decoding. In our closed-loop experiments, 
the adaptive decoder changed tuning model coefficients, baseline firing 
rates, and tuning model noise (see Figure 6). Though we cannot be certain 
that changes in tuning found by the method were not the result of recording 
instability, our results support the existence of changes in neuronal inputs 
observed by others and attributed to various reasons. 


5.2 Static Versus Adaptive Decoder. Adaptive decoding, while more 
complex than static decoding, offers the following advantages. If neuronal 
tuning changes during motor learning, an adaptive decoder could exploit 
the changes more than a static decoder, for example, by assigning more 
importance to neurons, which increase in tuning depth. If neurons change 
in tuning in a manner unrelated to learning, a static decoder’s accuracy 
would suffer, while an adaptive decoder could compensate. If the record- 
ing system degrades temporarily or permanently, an adaptive decoder can 
adjust the importance given to affected neurons to mitigate the effects of 
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noise. Coadaptive learning (Taylor et al., 2002), where neural circuitry and 
the decoder adapt to each other simultaneously, decreases training time 
and improves accuracy. An adaptive decoder allows long-term coadaptive 
learning, which may give larger improvements in control accuracy from 
practice than if practice was conducted with a static decoder. 

Ganguly and Carmena (2009) asserted that a nonchanging BMI decoder 
may be sufficient for long-term control accuracy, as demonstrated by BMI 
control using a static Wiener filter over 19 days. They found that motor cor- 
tical neurons initially changed their preferred direction and tuning depth. 
As neurons’ tuning stabilized, BMI control performance plateaued. They 
reported that even small changes to decoder parameters or loss of one or 
two neurons could harm control accuracy. Unlike our study, Ganguly and 
Carmena used only a subset of neurons, picked using criteria for spike 
waveform stability. This approach limits the communication bandwidth 
(which we loosely define as the aggregate recording quality, that is, the 
total number of recorded neurons weighted by how well each is recorded) 
of the BMI because neurons with useful tuning, but putatively unstable 
waveforms, may be ignored. Furthermore, if recordings for some of the 
putatively stable neurons worsen, control accuracy could suffer. Ganguly 
and Carmena also reported that BMI control accuracy with a static decoder, 
as measured by percentage correct and time to target, increased initially 
and plateaued after about 10 days of practice. In comparison, the control 
accuracy in our study, as measured by signal-to-noise ratio of the cursor 
trajectory with respect to the target trajectory, decreased slightly over 29 
days. The difference in trends may be due to the following factors. 

The performance metrics and behavioral tasks may not be comparable 
between the studies. The center-out task does not prescribe trajectories, un- 
like the pursuit task. The signal-to-noise ratio of the BMI cursor trajectory 
should have a higher level of saturation than the percentage of trials suc- 
cessful and time to reach. That is, improvement to control accuracy, which 
already saturates the percentage correct and time-to-target metrics, can still 
be detected by the SNR of the trajectory. Thus, a BMI user may maintain 
the same (saturated) percentage correct and time-to-target values, while 
decreasing in the SNR of the trajectory. 

Our decoder, parameter fitting procedure, and the use of a much larger 
neuronal population may explain our high initial control accuracy and 
slight decrease in accuracy over time, as opposed to the very low initial 
accuracy and improvements over time reported by Ganguly and Carmena 
(2009). Our regularized initial model fit, combined with a more sophisti- 
cated decoder (Li et al., 2009, for a comparison) and larger population size, 
may have decreased the possible benefit of extra practice with BMI control. 
Thus, control accuracy did not increase over time in our study. The simpler 
decoder, unregularized parameter fitting procedure, and small population 
that Ganguly and Carmena used could be the reason that initial control was 
poor and practice over several sessions could improve control. 
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The time span of this study was longer than that of Ganguly and Car- 
mena (2009). Combined with the fact that we did not pick neurons for 
waveform stability, this means that neuronal recordings may have de- 
graded more in our study. The recorded waveforms of some neurons may 
have changed, causing an increase in spike-sorting error and a decrease 
in available communication bandwidth. Although our adaptive decoder 
could adapt to tuning changes by changing the tuning model and handle 
increased noise by decreasing the relative contribution of affected neurons, 
the overall decrease in available communication bandwidth could be the 
cause of the slight decrease in control accuracy. In contrast, our static de- 
coder did not compensate for recording degradation and thus performed 
much worse. Reliable detection and attribution of spikes should reduce the 
large contrast between static and adaptive decoders by reducing the need to 
adapt. 


5.3 Adaptive Filtering. Adaptive filters deal with estimation of a time 
series with changing parameters, which describes the adaptive decoding 
problem. Generally an adaptive filter treats the unknown, changing pa- 
rameters as variables to be estimated. Two established approaches are joint 
filtering and dual filtering. Joint filtering (Kopp & Orford, 1963; Cox, 1964) 
augments the unknown parameters into the state space and performs es- 
timation on the augmented state. The augmentation generally results in a 
nonlinear system. The dual filtering approach (Nelson & Stear, 1976) uses a 
separate parameter filter whose state space holds the unknown parameters 
of the system. The parameter filter operates in lockstep with the state filter 
(the filter that estimates the state variables of the system), and each filter 
uses the output of the other as the truth. 

Another approach for identification of parameters is to perform 
batch-mode computations. Shumway and Stoffer (1982) used the 
expectation-maximization algorithm to identify parameters of a linear 
system. Beal (2003) presented a variational Bayesian treatment of the 
expectation-maximization identification approach; the Bayesian extension 
incorporated priors on parameters, thus allowing Bayesian parameter 
updates. 

An advantage of the batch mode update approach is that one can ap- 
ply Kalman smoothing to the filter output before using it for updating. 
Kalman smoothing improves the accuracy of the data used for parameter 
updates and allows estimates of movements later in time to be used to 
improve estimates of movements earlier in time. Since the backward pass 
of the Kalman smoother iterates backward in time, it is not possible to 
use Kalman smoothing with joint or dual filtering. Furthermore, the batch 
mode operation allows updates to occur asynchronously, taking as much 
time as needed and as few computational resources as desired. The iterative 
procedure in variational Bayesian regression may be stopped early to trade 
accuracy for speed. 
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5.4 Adaptive Decoding. Some approaches to learning initial decoder 
parameters may be applicable to learning changes in parameters. Tay- 
lor et al. (2002) used a coadaptive framework for fitting decoder param- 
eters using assumed desired movement trajectories. Musallam, Corneil, 
Greger, Scherberger, and Andersen (2004) assumed knowledge of correct 
targets in a target selection task for fitting decoder parameters. Wolpaw and 
McFarland (2004) used knowledge of the correct target to learn parameters 
for an EEG decoder. Gage, Ludwig, Otto, Ionides, and Kipke (2005) used 
the true tone frequency to learn parameters for a decoder in a coadaptive 
frequency-matching task. Wahnoun et al. (2004, 2006) and Hochberg et al. 
(2006) used a visual-following paradigm to fit decoder parameters. In the 
visual-following paradigm, the BMI user is instructed to imagine moving a 
cursor that is actually controlled by the computer or a technician, and the 
cursor trajectory is used to fit decoder parameters. DiGiovanna, Mahmoudi, 
Fortes, Principe, and Sanchez (2009) used reinforcement learning to fit 
parameters for a decoder in a two-target robot arm control task. Darmanjian 
and Principe (2009) developed a linked-mixtures hidden Markov model for 
unsupervised clustering of movement states but used Wiener filters fit from 
example hand movements to reconstruct trajectories. 

Some studies have focused on how to update decoder parameters. Wu 
and Hatsopoulos (2008) proposed a real-time update technique for linear 
decoders using recorded hand movements. Shpigelman et al. (2009) demon- 
strated real-time updating of decoder parameters using the assumption that 
the BMI user tried to move the cursor closer to a known target. Rotermund, 
Ernst, and Pawelzik (2006) proposed a stochastic search update method 
that uses a hypothetical performance evaluation signal recorded from the 
brain. 

Unlike the methods mentioned above, our adaptive decoder does 
not require knowledge of the desired movement. However, it still 
offers competitive accuracy. Wu and Hatsopoulos (2008) reported 
0.0022 cm? and 0.0053 cm? per trial reduction in mean squared error in 
two sessions when updating the model parameters of a Kalman filter using 
recorded hand movements. Using the trial length and static Kalman filter 
accuracies they reported, the MSE reduction translates to an improvement 
of 0.016 dB/min and 0.038 dB/min. In comparison, our adaptive decoder 
produced a 0.036 dB/min mean improvement for the joint formulation and 
0.033 dB/min mean improvement for the factorized formulation without 
using hand movements. When we used hand movements for updates, the 
improvements were 0.078 dB/min (joint) and 0.082 dB/min (factorized). 

Some studies have focused on adapting parameters without knowledge 
of desired movements. Sykacek and Roberts (2002) and Sykacek, Roberts, 
and Stokes, (2004) devised a variational Bayesian classifier for EEG sig- 
nals that updates parameters without knowledge of the desired classi- 
fication. Additionally, their method inferred the parameter change rate. 
Eden et al. (2004) and Srinivasan et al. (2007) demonstrated the ability of a 
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point-process filter to adapt to tuning model parameter changes using a joint 
filtering approach in simulation. Rickert et al. (2007) showed that joint filter- 
ing using an unscented Kalman filter can adapt to changing parameters in 
simulation. Liao et al. (2009) presented a dual filtering solution for adaptive 
decoding of myoelectric recordings. Gilja et al. (2009) updated the tuning 
model parameters of a Kalman filter using self-training. Unlike our adaptive 
decoder, their method replaced old parameters with new ones. This replace- 
ment approach does not take into account the information held in the old 
parameters. 

In our adaptive decoding problem formulation, since we allow all de- 
coder parameters to change, there is no fixed relationship between neu- 
ronal recordings and desired movements. There is no guarantee that a 
decoder would converge back to the correct answer if it gets off-track (in 
either movement estimates or parameter estimates) without knowing the 
real desired movements or real tuning parameters. There is no theoretical 
guarantee that the adaptive decoder finds the real tuning parameters once 
they change, though it empirically performs better than not adapting at all. 
If the desired movements are available for use in updates, there are con- 
vergence guarantees from the recursive system identification literature (see 
Ljung & Soderstrom, 1987). Previous studies have shown successful adap- 
tive decoding using hand movements or assumptions on desired move- 
ments (Wu & Hatsopoulos, 2008; Shpigelman et al. 2009). Although there 
are no convergence guarantees, previous studies have shown successful 
adaptive decoding in simulations where the real desired movements were 
not used (Eden et al., 2004; Srinivasan et al., 2007; Rickert et al., 2007). In 
clinical practice, if the adaptive decoder gets off-track, desired movements 
(e.g., target trajectory in a pursuit task practice session) can be used for a 
few updates to reset the tuning model parameters. 

The accuracy of adaptive decoding improves with the number of tuned 
neurons in the population. This is because the decoder combines observa- 
tions from the entire population, including any recently changed neurons 
(for which it has incorrect parameters). The larger the ratio of correct signal 
(from the unchanged neurons) to noise (from the changed neurons), the 
more accurate the decoder output. More accurate decoder output leads to 
more accurate parameter updates for the changed neurons. Thus, a larger 
recorded population will allow adaptation to more changing neurons or to 
larger changes. 

We did not provide specifics for the question of what initial tuning 
model priors and tuning model transition parameters (e.g., the model drift 
parameter) to use in a clinical setting. A reasonable and practical approach 
could be to use the range of values that work well with animal experiments 
and allow adjustment by the clinician or user. Another approach is to iden- 
tify the parameter settings using a more sophisticated model, similar to the 
work of Sykacek and Roberts (2002), Sykacek et al. (2004), and Ting et al. 
(2005). This challenging problem is left for future work. 
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During the course of closed-loop experiments, we noticed that the noise 
variance for some neurons approached 0. This was due to decreases in their 
firing rate variance. This led to near 0 predicted movement variances, which 
caused numerical problems in the Kalman smoother in a few updates. When 
this occurred, we simply skipped the Kalman smoothing step. Exploring 
the use of a lower bound on noise variance or the use of an inversion-free 
smoother are possible topics for future work. 


Appendix: Conversion Between Signal-to-Noise Ratio 
and Correlation Coefficient 


We derive the approximate conversion of signal-to-noise ratio to Pearson’s 
correlation coefficient below. We denote the desired signal by x; and the 
predicted signal by y;, where i indexes on the n data points. We use x and 
y to denote the means, and we use s, and s, to denote the sample standard 
deviations. 

We start with a definition of Pearson’s correlation coefficient r: 


i! “ %—-*\ fH -y 
r= = )( : ) (A.1) 


i=l x y 


Using the assumptions of zero mean and equal variance for x and y 


ETT (A.2) 


where v is the shared sample variance of x and y. Next, we examine the 
definition of mean squared error: 


n 


1 
MSE = — Y°(x; — y)) (A.3) 


i=1 


We expand the squared term in the summation and use the fact that x 
and y have zero mean to recognize the population variance terms in the 
results: 


= I “ 2) 2 
MSE = dX, (x7 — 2xy,+y;), (A.4a) 
1 n 
MSE ~ 2v — — > 2x}. (A.4b) 


i=1 
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The approximation is due to the substitution of sample variance for pop- 
ulation variance. Next, we note the resemblance of the summation to the 
equation for r in equation A.2 and substitute 


MSE ® 2v — 2ur. (A.5) 


This is another approximation, due to the substitution of +, for +. Next, 
we substitute our approximate MSE into the equation for SNR: 


Uv 
SNR,jp = 10 - logig (scx) (A.6a) 
Vv 
SNR jp ~10- logy (<5) ; (A.6b) 
al 
SNR,p © 10- logy, (> - =) (A.6c) 


Finally, we solve for r in terms of SNR to arrive at equation 2.3. 
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High-Side Digitally Current Controlled 
Biphasic Bipolar Microstimulator 


Timothy L. Hanson, Bjérn Omarsson, Joseph E. O’Doherty, Ian D. Peikon, Mikhail A. Lebedev, and 
Miguel A. L. Nicolelis 


Abstract—Electrical stimulation of nervous tissue has been 
extensively used as both a tool in experimental neuroscience 
research and as a method for restoring of neural functions in 
patients suffering from sensory and motor disabilities. In the 
central nervous system, intracortical microstimulation (ICMS) 
has been shown to be an effective method for inducing or biasing 
perception, including visual and tactile sensation. [CMS also holds 
promise for enabling brain—machine—brain interfaces (BMBIs) 
by directly writing information into the brain. Here we detail the 
design of a high-side, digitally current-controlled biphasic, bipolar 
microstimulator, and describe the validation of the device in vivo. 
As many applications of this technique, including BMBIs, require 
recording as well as stimulation, we pay careful attention to isola- 
tion of the stimulus channels and parasitic current injection. With 
the realized device and standard recording hardware—without 
active artifact rejection—we are able to observe stimulus artifacts 
of less than 2 ms in duration. 


Index Terms—Artifact, cortex, microstimulation, suppression. 


I. INTRODUCTION 


UMEROUS studies have shown that electrical micros- 
timulation of neural circuits by injecting small currents 
from an electrode tip into the nervous tissue may evoke a variety 
of effects that are often similar to the functional contribution of 
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the stimulated area [1]. In particular, microstimulation can pro- 
duce or bias sensations [2]-[8]. This property of bioelectrical 
stimulation has attracted the attention of neural engineers as the 
key component of sensory neural prosthetics for the restoration 
of sensation in patients suffering from sensory disabilities. 

Microstimulation has been introduced as a sensory loop in 
brain-machine-brain interfaces (BMBIs), systems that both 
translate brain activity into commands to artificial actuators and 
deliver information to the brain in the form of microstimulation 
of sensory areas. In experiments conducted in our laboratory, 
we utilized temporal patterns of cortical microstimulation to 
create an artificial somatosensory input for the BMI that enacted 
arm reaching movements[9]. More recently, our laboratory has 
demonstrated closed-loop BMBI control for delivering artificial 
texture feedback through ICMS[8], where precise, rapid, and 
low-artifact ICMS control was essential to give feedback to the 
monkey without interfering with recording periods. Here, we 
describe our custom multichannel microstimulator that enabled 
these BMBIs. 

Intracortical microstimulation (ICMS) poses a number of 
exacting requirements on the experimental system. ICMS 
requires relatively high voltages (50-100 V) when high 
impedance (0.5—2 MQ) electrodes are used [10]; [11]; [12]. 
Furthermore, the useful stimulation waveform durations are 
usually very short (10 ys to 100 ys) [2], [7], [11], [12], [13]. 
The waveform most frequently used is a charge balanced, 
symmetric, biphasic stimulation pulse, where a square cathodic 
pulse is followed by an anodic pulse of equal amplitude [14]. 
Although the second pulse can reduce the excitation of the 
tissue, damage to the electrode and tissue is prevented by this 
charge balancing pulse. The impairing effect of the second pulse 
can be partially counteracted by a short (100 js) delay between 
the two pulses [15]). Due to the variable electrode to tissue 
impedance, and response of the tissue to charge, cortical stimu- 
lation is usually performed with constant current sources [16]. 
In terms of safety, the range of figures quoted here—100 pus 
pulses of 100 zA/phase—equates to 10 nC/phase, well within 
the regions described by [17]. Charge density using our 65 wm 
electrodes is 300 C/cm?/phase, which is also within the 
safety limits proposed by [18]. 

The current, voltage and timing requirements are not diffi- 
cult to achieve with modern electronic systems, and many com- 
mercial products are available that satisfy the conditions. How- 
ever, for advanced ICMS studies, the complexity of the stim- 
ulus train and number of electrodes enforce more rigorous re- 
quirements on stimulation technology. There is immediate need 
for advanced microstimulation systems capable of operating as 
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ICMS feedback systems in BMBIs [7], [9], [19], [20]. These 
systems require microstimulation in multiple sites in the brain, 
using multiple electrodes and complex spatiotemporal stimula- 
tion patterns [7], [9], [21], delivered at low latency to enable 
closed-loop control [8]. It is critical for these systems to induce 
artifacts of minimal duration to adjacent recording sites to max- 
imize recording quality in BMBIs. 

Our stimulator is high-side current controlled, which nearly 
eliminates the charge and discharge current of stray capacitance 
between each isolated channel and animal ground. High-side 
means that the compliant current source is tied to the voltage 
rails, not ground; with a low-side current source, the current 
regulator develops voltage across it depending on electrode 
impedance, voltage that is presented to the stray capacitance 
of the microstimulator and experimental wiring. Since currents 
associated with charging and discharging voltage offsets from 
stray capacitance must necessarily go through the stimulation 
electrodes, and ultimately through the ground that connects the 
animal to wired recording systems, voltage offsets should be 
minimized for both accuracy and artifact avoidance. High-side 
current control is, to the best of our knowledge, unique to this 
design. Further artifact suppression can be achieved through 
close-proximity bipolar biphasic stimulation, which restricts 
the spread of electric field within the tissue while remaining 
efficacious [7]. Finally, the described stimulator is highly 
flexible: both the current and duration of both phases of the 
biphasic stimulus waveform can be controlled continuously 
from a computer with tight synchrony. This computer control 
permits stimulus trains of arbitrary complexity to be enacted in 
real-time as needed for a BMBI. 

It should be emphasized that the construction and testing of 
this device was done explicitly for a tethered research environ- 
ment—that is, where the microstimulator is not mounted on or 
in the experimental animal—as no commercially available mi- 
crostimulators were available to permit high temporal fidelity, 
high flexibility (any pattern can be commanded), high compli- 
ance, good current resolution, good channel isolation, minimum 
parasitic current injection, and a flexible number of channels. 
Full specifications of the described device are listed in Table I. 
Given these design criteria the size of the device is large and 
consumes 300 mW per channel. In comparison, other research 
into microstimulators such as the clinically-targeted Bion [22] 
are much smaller and lower power, but correspondingly much 
more expensive, less flexible due to their high level of integra- 
tion, and not matched for the high compliance voltage required 
by our electrodes. Integrated ASIC microstimulators such as [23] 
and [24] offer compliance of 11 V or less, which is insufficient to 
drive current through high-impedance (more than 100 k2) elec- 
trodes; full systems described in [25] offer insufficient latency 
and bandwidth for our experimental requirements. Finally, the 
current-mirror topology of integrated microstimulators does not 
guarantee low leakage current, which is critical as de current will 
gradually erode the tips of electrodes and damage neural tissue. 


Il. METHODS 


A. Microstimulation System 


The full microstimulation system is comprised of four prin- 
cipal elements: high-level control programs (web interface, 


TABLE I 
MICROSTIMULATOR SPECIFICATIONS 
Parameter Value 
Compliance 50 V (160 V max) 
Current 800 pA 
Current resolution 0.2uA 
Output impedance >10 MQ 
Leakage current 300pA 
Parasitic current injection 30 pC/pulse 
Charge imbalance -0.094 +0.56 nC 
Channel isolation 15 pF 
Timing resolution 10 ps 
Timing latency 5.4ms+180 ps 
Power 300mW/channel 
Size 13cm x 2.5 cm/channel 
Channel count 15 (bipolar/monopolar) 
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Fig. 1. Schematic overview of software control of the microstimulator. 


UDP server), a low-level driver/DMA control program, stim- 
ulus isolation channels, and the stimulation electrode array 
(including lead system). Software control of the microstimula- 
tion system is depicted in Fig. 1. 


B. Software Control 


Interactive control of the microstimulator is through a custom 
driver program. This program, which runs on a dedicated com- 
puter Linux running a low-latency kernel, services a free run- 
ning National Instruments PCI-6533 card, which continually 
outputs 16 or 32 bits of digital data at a clock rate of 100 kHz. 
These digital data consist of multiplexed control pulses and se- 
rial peripheral interface (SPI) commands to set the channel cur- 
rent. Data to be output are read by the PCI-6533 from a cir- 
cular DMA buffer, the address of which is acquired through a 
custom Linux kernel module. The driver maintains the DMA 
buffer via a set of watermarks, placement of which depends on 
the speed of the computer and its peripheral subsystem. To pre- 
vent underflow, typically this is less than 200 samples, which 
translates to a delay of 2 ms at the 100 kHz clock rate. Between 
DMA servicing cycles— when the watermark criteria are satis- 
fied—the driver program interprets control commands written 
to a common memory-mapped file. 

Microstimulation commands can either be issued through a 
python-based web interface, or from a UDP server. The latter 
allows the stimulator to be controlled through Matlab or from 
other experimental software, such as the BMI software devel- 
oped in our lab. The web interface allows easy user control of all 
parameters of stimulation, e.g., current amplitude, pulse width, 
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Fig. 2. Schematic overview of one stimulator channel. 


frequency, and secondary frequency from any networked com- 
puter. The present software implementation allows the stimula- 
tion pattern to be a single pulse, a continuous train of pulses, a 
pulsed stimulation train, a doubly-pulsed stimulation train cor- 
responding to three periods and duty cycles [12], or a stochastic 
stimulus train derived from a gamma distribution. More com- 
plicated stimulus programs can easily be added to the driver 
program to provide more sophisticated feedback. In the experi- 
ments described in [8], where 50 ms of BMI decoding alternated 
with 50 ms of microstimulation, we minimized kernel overhead 
and tightened DMA watermarks so that the latency from BMI 
command to stimulation, including UDP transmission over eth- 
ernet, was 5.4 ms+180 ys (mean+standard deviation). This 
low-variance latency permitted precisely timed ICMS feedback. 


C. Stimulus Isolation Units 


Four stimulator channels are assembled on a circuit board, 
providing independent and isolated monopolar or bipolar stim- 
ulation per electrode. Four of these boards can be stacked and 
serviced by one PCI-6533, for a total of 15 stimulation chan- 
nels.! An overview of a single channel is shown in Fig. 2. 

There are four primary elements to each channel: the power 
supply, digital-to-analog converter (DAC), bipolar current reg- 
ulator, and an isolated monitoring circuit. The isolated power 
supply uses two series 1 W miniature dc-dc converters to pro- 
vide 40-160 V for high compliance/high impedance electrodes. 
An additional 1 W dc—de converter is used to supply 5 V for 
the DAC and other circuitry. Current is commanded via the 
SPI DAC, with separate commands for the anodic and cathodic 
phases; the command signals for each DAC (clock, data, chip 
select, load output) are multiplexed with the anodic and ca- 
thodic pulse commands so only four signals need pass through 
the magnetic isolator. Stimulation is enabled with the inverted 
DAC chip select signal; current amplitude can hence be changed 
whenever stimulation pluses are absent. Chip select and DAC 
load output signals are common among all channels, thereby 
requiring the total number of digital control signals to be two 


1The PCI-6533 has 32 digital input/output channels, and each channel re- 
quires two lines plus two common signals for all channels, hence only 15 chan- 
nels can be controlled from one PCI-6533 card. 
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plus twice the number of channels; in turn this means currents 
on all channels must be set (but not necessarily changed) at the 
same time. 

The core of the stimulation unit is a voltage-feedback high- 
side current source, shown in Fig. 3. It is vital that the current 
source is high-side, as the topologically simpler low-side con- 
trol alternative necessitates that during the anodic and cathodic 
phases the current needed to charge the stray capacitance from 
one channel to ground must pass through the output electrodes. 
Note the current source is a symmetric H-bridge, so for clarity 
only half is fully shown in the figure. Briefly, each half receives 
two inputs: a voltage command from the DAC into Vdac_A, 
and an inverted pulse command on Ain. Transistors Q2 and 
Q9 act as common-collector amplifiers to detect differences be- 
tween voltage across the sense resistor, RI, and Vdac. Tran- 
sistors Q4-Q7 act as a differential amplifier to set the current 
through Q11 and Q15. The latter directly controls the current 
into the opposite electrode in the pair, while the former supplies 
current to turn on the lower leg of the H-bridge, Q12. Note that 
base current into Q12 also goes through the sense resistor R1, 
so the current command is off by a fraction set by the ratio of 
resistors R2 and R3 * Gg 5. This offset is measured below and 
corrected in software. Capacitor C1 controls the slew rate of the 
Q4-Q7 pair, and prevents feedback oscillations. Finally, Q3 and 
Q10 serve to turn off the output stage when the control signal 
goes high; Q10 is essential for draining base charge on Q12 and 
returning the output to a high impedance condition. Buffer U1 
serves to delay this command to Q10, allowing Q12 to remain 
on for about 4 ys longer so as to discharge any residual charge 
on the electrodes, and hence between the animal ground and 
isolated ground. As mentioned before, this is to minimize arti- 
fact on neighboring recording electrodes. Transistor Q3 is sat- 
urated on when output is disabled, which turns Q5 on through 
the current mirror, hence setting the base current of Q15 to zero; 
this means that stimulator leakage current is dominated by the 
off-state current mismatch through Q15 and Q12, which is mea- 
sured to be less than 300 pA. It was chosen to make the output 
high-impedance when off rather than short-to-ground to further 
minimize noise injection. 

For safety, diodes D4, D5 and their mirrors serve to protect 
the output stage from ESD damage along with resistors R1 and 
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Fig. 3. Detailed schematic of one-half of the voltage-controlled high-side current source. This topology is mirrored to provide a bipolar current source from one 


voltage supply. The mirror axis is shown by the dotted line. 
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Fig. 4. Isolated monitoring circuit. 


R2. If this topology is used in a clinical setting, further protec- 
tion will need to be added, as failure-to-short of any transistor 
can tie one half of the H-bridge to a supply rail, and failure of 
two transistors can permit unregulated dc current through the 
electrodes. In the implemented boards, all transistors are rated 
at 185 V or higher, while the supply voltage is typically 50 V; no 
semiconductor failures have observed in several years of oper- 
ation, though one channel in one stimulator was disabled prior 
testing due to internal PCB delamination. This is best fixed by 
simply using a thicker medical-grade PCB. Finally, While there 
are no hardware limits on the duration of delivered current, we 
have observed no software failures leading to de current applied 
to the animals. Transistor bias currents limit total current deliv- 
ered per channel to 800 pA. 

The isolated monitoring circuit stage is shown in Fig. 4. 
Blocks U1 and U2 contain one infrared LED and two matched 
photodiodes. Opamps O1 and O2 regulate the LED current to 
match the photodiode current to that from inputs Aout’ and 
Bout’ (see Fig. 3); this mirrors the current onto the non-isolated 
photodiode pairs. Opamp O3 converts the resulting nonisolated 


iso 


bipolar photodiode current to a voltage, and opamp O4 works 
to null de differences due to photodiode imperfections. In turn, 
06 drives indicator LEDs, and O5 the final output, which can 
be viewed on an oscilloscope or digitized. Because the current 
command and resultant voltage are known, this circuit allows 
electrode resistance to be easily calculated. 


D. Electrodes 


The electrodes used in testing this system were fabricated 
in-house at the Duke University Center for Neuroengineering 
(DUCN). The electrodes work well for both chronic multisite 
neural ensemble recordings and for stimulation. The DUCN 
microelectrode arrays consist of tungsten or stainless steel 
microwires with teflon, SML, or HML insulation. The overall 
diameter of the microwires is in the range of 25-65 jum, 
and the separation of electrodes in the array is the range of 
200-1000 zm, center to center, depending on the brain area 
targeted, the animal species, and the experimental protocol 
[26]. For all testing and validation use of the system reported 
here, we stimulated between two of electrodes chosen from an 
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Fig. 5. Photograph of a populated PCB featuring four bipolar channels; input 
is on the lower left, output on the right. 


arrays constructed with stainless steel microwires and HML 
insulation. 


E. Assembly and Testing 


Electrical schematics were designed in the open-source 
Kicad software suite; the PCB was designed in Kicadocaml 
then manufactured by Imagineering, Inc. (Elk Grove, IL). We 
elected to use entirely discrete components in this device for 
manufacturing ease and cost; the current regulator and other 
parts could be integrated in a high-voltage bipolar process. 
The smallest components placed were 0402 chip resistors and 
capacitors, though the majority of the PCB is populated with 
0603 scale resistors and transistors, only on the top of the board; 
this leads to a total PCB area for 4 channels of 187 cm? witha 
mass of 120 g. As mentioned in the introduction, this is much 
larger than comparable microstimulator ASICs designed for 
implantation, which are less than 1 cm?, but the device meets 
its intended research purposes. Four PCBs can be stacked to 
obtain 15 stimulation channels. A photograph of the assembled 
PCB is shown in Fig. 5. 


IJ. RESULTS 


A. Bench-top Testing 


The system was first tested using resistive loads to verify 
proper circuit performance and timing. Fig. 6 shows the output 
of the simulator through a 100 kQ resistor and the resultant 
isolated monitoring output. The latter has a voltage attenua- 
tion of 25 to allow headroom for measuring large-amplitude 
signals or high-impedance electrodes. In practice, the wave- 
forms have much lower slew rate due to the capacitance of the 
wire between electrode and microstimulator and the nonlinear 
charge/discharge profile of the electrode—tissue interface. 

The implemented system is capable of a current range of 
0-400 yA or 0-800 A, depending on the DAC output scaling. 
Output currents larger than this will require different bias cur- 
rents in the bipolar current regulator, hence a few component 
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Fig. 6. Top: output of microstimulator through a 100 k©? resistor, current set at 
200 yA, anodic and cathodic phases set to 100 jus, delay between pulses 50 jus. 
Bottom: simultaneous output of isolated monitoring circuit. 
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Fig. 7. Rise and fall times for anodic and cathodic phases. Slight differences 
may be due to the fact that this was measured with one oscilloscope probe, hence 
the anodic pulse had to charge stray capacitance whereas the cathodic did not. 


values would need to be changed. The stimulator features a mea- 
sured maximum voltage compliance of 160 V, and an output par- 
allel resistance of greater than 10 MQ). To characterize the slew 
rate of the current source, rise and fall times of the pulse output 
were measured across a 118 KQ resistor for varying currents. 
Rise and fall times are defined as the time for the voltage to in- 
crease from 10% to 80%, and to fall from 80% to 10% of the 
output amplitude, respectively. As can be seen in Fig. 7, the rise 
and fall times ranged from 6—20 ps for output voltages of S—38 
V, with longer rise and fall times at higher output amplitudes. 
This corresponds to a slew rate of 5 V/ys at larger amplitudes; 
this may be decreased by increasing the value of capacitor Cl 
in Fig. 3. 

As mentioned above, the current output is less than the com- 
manded output current by the base current of transistor Q12 in 
Fig. 3. This current was set conservatively in the present imple- 
mentation to allow higher output currents, hence must be cor- 
rected in software. To calibrate the device, we tested the output 
across 97, 120, 180, 240, 270, 330, and 470 k{© resistors, with 
current amplitudes of 50, 75, and 100 yA. Fig. 8 shows the re- 
sult of this calibration, in which the trend line, found using linear 
regression, has slope of 0.745. The pulses measured in this test 
were deemed to have started when the voltage reached 50% of 
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Fig. 8. Expected versus actual pulse amplitude for several current settings and 
resistive loads. 
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Fig. 9. Left: distribution of measured pulse widths for a 100 js command 
pulse. Right: distribution of charge imbalance for pairs of anodic-cathodic 
pulses. 


peak amplitude, and to have ended when the voltage dropped to 
50% of peak amplitude. The output voltage amplitude is defined 
as the mean voltage during that interval. 

Using these criteria, we also measured the distribution of 
pulse widths, shown in Fig. 9. For a 100 ss command pulse, 
the distribution of measured pulse widths was found to have 
a mean of 92 + 1.3 ys. The charge balance of each stimu- 
lation pulse was calculated, and the distribution is shown to 
the right. The mean charge imbalance for commanded pulses 
was—0.094 + 0.56 nC. For comparison, the charge delivered 
in each phase is 5—10 nC, so this imbalance is only on the order 
of 1%-2%. 


B. In Vivo Testing 


To characterize the stimulator in intended application, we ex- 
amined the effects of ICMS in a rhesus monkey chronically im- 
planted with multielectrode arrays bilaterally in motor and sen- 
sory cortices. In the first of these tests, we applied stimulus trains 
consisting of 150 ys long, 100 wA anodic and cathodic pulse 
pairs separated by a delay of 25 ys. Fifty of these pulse pairs 
were applied with an inter-pulse interval of 10 ms to electrodes 
spaced at 1 mm in the arm representation of the left primary 
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Fig. 10. Three example voltage traces recorded from different electrodes in 
right M1 arm region of a rhesus macaque. Traces show varying amounts of 


stimulus artifact and simultaneous extracellular action potentials, as sorted in 
the Plexon software, indicated with red asterisks. 


motor cortex (M1). Simultaneously we recorded using a Plexon 
Inc. (Dallas, TX) Multi-acquisition processor (MAP) from elec- 
trodes implanted in the arm region of M1 of the opposite hemi- 
sphere, using standard spike (passband: 170 Hz-8 kHz, gain: 
1000) preamplifier boxes and custom headstages with a gain of 
8 and one high-pass pole at 560 Hz. 

In Fig. 10, trace A was recorded from a blunt-cut 50 pm 
Teflon-insulated stainless electrode at +15.5 mm lateral the 
midline, +4.5 mm rostral the intra-aural line, or 32 mm from 
the center of the dipole created by the stimulating electrode. 
Trace B is from the same type of electrode, at +12 lateral, 
+10 rostral, making it 29.2 mm from the stimulating dipole; 
trace C is from a sharpened HML-insulated 65 j:m stainless 
electrode +15 lateral, +6 rostral, hence 31.5 mm from stim- 
ulating dipole. Despite the fact that all three recordings were 
from about the same distance from the stimulating dipole, 
the top trace shows no stimulation artifact, the middle shows 
a small stimulation artifact, and the bottom shows a large 
stimulation artifact. The differences are likely due to different 
referencing for the three channels—channels A and B were 
referenced to electrodes in the microelectrode array function- 
ally identical to the recorded electrode, while channel C was 
referenced to animal ground, which is connected to T-bolts im- 
planted through the scull. In the Plexon preamplifier, reference 
channels are subtracted from individual electrode channels to 
cancel out common-mode noise; it can work well though better 
methods exist [27]. 

To further test the degree to which microstimulation inter- 
feres with neural recording, we needed to increase the artifact 
on channels A and B shown in Fig. 10. As increasing the spacing 
between electrodes increases the size of the associated electric 
field, we made the distance between the stimulating electrodes 
as 5.6 mm, as large as possible within one microelectrode array 
in the chronically implanted monkeys. Fig. 11 shows the re- 
sulting traces, which indicate accordingly that the artifact has 
increased as compared to Fig. 10. 

Importantly, neuronal spikes are clearly still visible in be- 
tween stimulus artifacts on Fig. 11 traces A and B. Trace C 
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Fig. 11. Three electrode traces recorded from right M1 arm region of a rhesus 
macaque. Stimulus artifact resultant 100 jus, 100 2A anodic and cathodic pulses 
separated by a delay of 100 pus. 


shows a broader artifact that may occlude spikes. To accurately 
determine the width of these artifacts, we next recorded contin- 
uously for one hour, again from several microelectrodes located 
in the arm representation of M1 while stimulating the contralat- 
eral hemisphere with 100 jus, 100 yA anodic and cathodic pulses 
separated by a delay of 100 ys. These pulses were applied once 
a second (1 Hz) to two electrodes spaced at 1 mm, the same 
electrodes as in Fig. 10. 

Figs. 12 and 13 show the two extremes of the result of this 
test: Fig. 12 shows a channel that was very minimally effected 
by the stimulus artifact, while Fig. 13 shows the channel that 
was broadly affected. Fig. 12 is consistent with only minor in- 
terference with spike detection and sorting, and only for those 
spikes whose duration spans the stimulus pulse. Fig. 13, how- 
ever, suggests that at least one part of the amplifier chain, most 
likely the high-gain preamplifier, is driven to saturation for the 
duration of the artifact; it is unlikely that the headstage is driven 
to saturation, as then the result in Fig. 12 would not be pos- 
sible. These two channels, other than slight differences in dis- 
tance from the stimulating pair, are only different in their refer- 
encing—the channel in Fig. 13 was referenced to ground. These 
results show that it is possible to record with very minimal 
stimulus artifact with the described stimulation system, proper 
referencing, and centimeter spacing between stimulating and 
recording electrodes. Note that no stimulation artifact suppres- 
sion system was used. 

To further characterize the duration of the artifact for much 
closer stimulation-recording distances, we examined the stim- 
ulus artifact on 112 channels during a closed-loop stimulation 
task with a rhesus macaque. Of the 112 recording electrodes, 32 
were in motor cortex contralateral to stimulation, mean distance 
3641.7 mm from stimulating pair; the other 80 were ipsilateral, 
in the same array as the stimulating electrodes, mean distance 
2.7+1.2 mm. Plotting stimulus artifact duration by absolute dis- 
tance from stimulating pair did not yield any observable trends 
due to varying referencing and per channel gain, so the aggre- 
gate data is presented in Fig. 14. Stimulus artifact in contralat- 


337 


Firing rate (Hz) 


10} 1 


5 M 4 4 1 4 1 1 1 1 
-0.05 -0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.04 0.05 
Time relative to stimulation (s) 


(b) 


Fig. 12. (a) Rasterplot of spike times relative to the 1 Hz stimulation pulses; 
electrode and neuron were the same as shown in A of Figs. 10 and 11. (b) Peri- 
event time histogram (PETH) of the corresponding spike rate, binned at | ms. 
Artifact, while not visible in Fig. 10, is visible here, and is © 1 ms. 
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Fig. 13. (a) Rasterplot of spike times relative to the 1 Hz stimulation pulses; 
neuron was recorded from leg representation of right M1 at +4.5 lateral, 
+2 rostral from 65 jzm HML insulated stainless electrode, or 21 mm from the 
stimulating pair. (b) Peri-event time histogram (PETH) of the corresponding 
spike rate, binned at 1 ms. Artifact was 6.5 ms and was the longest of those 
recorded in this session. 


eral recording electrodes was 2.1 + 0.8 ms, and 5.8 + 2.3 ms 
for ipsilateral. Though the electrodes here are much closer to 
the stimulating pair than Figs. 12 and 13, this is short enough 
that spikes can be recorded between stimulus pulses on many 
channels, as can be observed from the aggregate firing rate in 
Fig. 14(a). 


C. Efficacy in Nervous Tissue 


To demonstrate the efficacy of the stimulator in activating 
nervous tissue, we performed stimulation of the right arm rep- 
resentation of M1, with two stimulating electrodes consisting 
of HML insulated sharpened 65 jum stainless steel, centered 
+15 mm lateral, +10 mm rostral. These electrodes were the 
deepest of the array and penetrated the cortex by + 3.4 mm. 
Stimulation pulses were 100 ys pulses of 75 4A, separated by 
50 ps; fifty of such pluses were applied for each trial at 100 Hz. 
The results of this protocol are depicted in Fig. 15. 
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Fig. 14. (a) Mean aggregate firing rate of 190 neurons recorded from 112 elec- 
trodes in left and right motor cortex from a rhesus macaque. Stimulus was 
groups of 8 150 ja pulses separated by 5 ms. Note blanking due to stimulus 
artifact and rebound excitation after stimulus group. (b) Boxplots of the stim- 
ulus artifact duration grouped by recording side. Box indicates 25% and 75% 
intervals, with the center line indicating mean duration, whiskers the distribu- 
tion of all data, and ’+-’ the duration of two outliers. 
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Fig. 15. EMG responses to stimulation of right M1 cortex. Top trace shows the 
timing envelope of the stimulus; below that: LWE = left wrist extensor; LWF 
= left wrist flexor; LB = left biceps brachii; LT = left triceps brachii; RWE = 


right wrist extensor; RWF = right wrist flexor; RB = right biceps brachii; RT 
= right triceps brachii. 


The microstimulation had a gradually increasing effect on 
EMG during the half-second of duration of pulses, as is evident 
in the Figure. More immediate responses and broader activa- 
tions (including muscles of the torso) were obtained with higher 
currents—up to 175 4A. No other effects, harmful or otherwise, 
were observed in the monkey. 


IV. CONCLUSION 


We have demonstrated the design of a multichannel high- 
side digitally current regulated microstimulator, and verified its 
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function on the bench as well as in an experimental animal. The 
design tries to minimize current injected as a result of parasitic 
capacitance between the microstimulator and animal ground, 
and as a result is capable of introducing only very short artifacts 
into simultaneous neuronal recordings—less than 1 ms in some 
configurations—which, when combined in low-latency control, 
is suitable for experiments where neural recording and micros- 
timulation are tightly interleaved. Our microstimulator offers a 
compliance of 50 V or greater for stimulating high-impedance 
electrodes with a very small leakage current of 300 pA or less. 
As such, the design is immediately applicable to experiential 
neurophysiology; indeed, in our lab we have been using three 
instances of the stimulator. 

Several improvements are suggested by the results. Other 
than further reducing parasitic capacitance, it would be useful 
to provide a greater range to the output current. For example, 
if the feedback resistor (R1 in Fig. 3) and bias currents are ad- 
justed, the same stimulator topology can easily be used on mus- 
cles for functional electrical stimulation (FES). An alternative 
to the present H-bridge design is a bipolar supply totem-pole 
configuration with current mirroring rather than feedback stabi- 
lization, so there is no need for a feedback resistor an its atten- 
dant voltage offsets; however, this design can be susceptible to 
leakage current and charge imbalance, both which can damage 
electrodes. Even though the microstimulation artifact is short, 
active or forward cancellation schemes may be desired to reduce 
the artifact to below the noise level. Perhaps the most pressing 
concern, however, is size—if microstimulation with simulta- 
neous recording is to be clinically applied, stimulation channels 
and controller will need to be integrated onto one chip, con- 
sume less than 1 mW, and likely share a common power supply 
rail with recording apparatus. We believe that the topology de- 
scribed herein represents a step forward in microstimulator de- 
sign for experimental use, that the lessons in minimizing artifact 
are instructive to other practitioners, and that the same compact 
topology may be integrated and miniaturized into a ASIC for 
future clinical use. 
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Chronic, wireless recordings of large-scale brain 
activity in freely moving rhesus monkeys 


David A Schwarz!?, Mikhail A Lebedev!”, Timothy L Hanson!?, Dragan F Dimitrov’, Gary Lehew!, 
Jim Meloy!, Sankaranarayani Rajangam!”, Vivek Subramanian”, Peter J Ifft”4, Zheng Li!, 
Arjun Ramakrishnan!, Andrew Tate!, Katie Z Zhuang”* & Miguel A L Nicolelis!4-6 


Advances in techniques for recording large-scale brain activity 
contribute to both the elucidation of neurophysiological 
principles and the development of brain-machine interfaces 
(BMIs). Here we describe a neurophysiological paradigm for 
performing tethered and wireless large-scale recordings based 

on movable volumetric three-dimensional (3D) multielectrode 
implants. This approach allowed us to isolate up to 1,800 neurons 
(units) per animal and simultaneously record the extracellular 
activity of close to 500 cortical neurons, distributed across 
multiple cortical areas, in freely behaving rhesus monkeys. 

The method is expandable, in principle, to thousands of 
simultaneously recorded channels. It also allows increased 
recording longevity (5 consecutive years) and recording of a broad 
range of behaviors, such as social interactions, and BMI paradigms 
in freely moving primates. We propose that wireless large- 

scale recordings could have a profound impact on basic primate 
neurophysiology research while providing a framework for the 
development and testing of clinically relevant neuroprostheses. 


Single-unit recordings from cortical neurons in awake, behaving 
monkeys were pioneered by Edward Evarts in the 1960s!, paving 
the way for more than five decades of experimental studies focusing 
on the neurophysiology of the nonhuman primate’s brain in action. 
Until the late 1990s, technological limitations allowed researchers 
to sample from just one neuron at a time or, in rare situations, from 
a few neurons simultaneously. The introduction of chronic multi- 
electrode implants!~* and the development of computer techno- 
logies for online information processing and analysis allowed 
several advancements in the field of primate neurophysiology, such 
as recording simultaneously from many neurons for extended peri- 
ods of time>®, extracting behavioral parameters from neural signals 
in real time’, and using brain-derived signals to control external 
devices through BMIs®?. Altogether, these developments trans- 
formed chronic brain implants into one of the most pervasive experi- 
mental approaches employed by systems neurophysiologists. 
Because only tens of cortical neurons can be routinely sampled 
simultaneously in macaques (a miniscule fraction of the hundreds 


of millions of neurons that form the monkey cortex!®), new neu- 
ronal recording methods are required to advance basic brain 
research, large-scale brain mapping and clinical translation of 
BMIs!!. Furthermore, tethered recordings in experimental animals 
have limited the range of natural behaviors that can be studied, 
particularly in nonhuman primates. The transition to using a low- 
power, implantable, wireless interface is imperative for the success 
of experiments aimed at recording large-scale brain activity in 
behaving primates. In response to this need, several multichannel 
wireless recording systems have recently emerged!?-!>. However, 
to date, no system has been shown to be scalable in the number 
of recording channels. 

Finally, considerable improvements in brain recording 
technology are needed before BMIs can become clinically relevant 
for helping severely disabled patients to regain mobility!!1617, 
For example, our estimates indicate that a BMI aimed at restor- 
ing limb movements may require 5,000-10,000 neurons recorded 
simultaneously!8, whereas 100,000 neurons will be needed to 
drive a BMI in charge of producing full-body movements!’. Here 
we introduce an integrated paradigm for chronic, multichannel, 
multisite, wireless large-scale recordings in freely roaming pri- 
mates. We report the first volumetric recording probes with 
thousand-channel capacity, evidence of close to 5 years of contin- 
uous recordings, and the first scalable wireless recording interface 
validated in naturally behaving, unrestrained monkeys. 


RESULTS 

Chronic multichannel implants 

Our results were obtained in eight adult rhesus monkeys (Table 1). 
Five monkeys received movable volumetric implants in mul- 
tiple cortical areas of both cerebral hemispheres. Additionally, 
we present data from three rhesus monkeys (monkeys I, G and Cl) 
implanted with previous-generation microwire arrays composed 
of fixed (non-adjustable) microelectrodes. We also show the latest 
version of our movable volumetric implants, called recording 
cubes (Fig. 1a). We built each of these cubes by first creating an 
array of polyimide guiding tubes, spaced at 1 mm apart (4 x 10 or 
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Table 1 | Implant summary 


Additionally, because our chronic cortical 


recordings have lasted for close to 5 years 


Units per Max. Max. 
Time with Max.  microwire simultaneous simultaneous in our monkeys (see below), itis likely that 
Channels implant units (global channels channels our implantation paradigm also maintains 
Monkey implanted (months) isolated average) (wired) (wireless) Refs. healthier cortical tissue. 
0) 1,792 13 1,874 0.522 512 N/A This work After the recording cubes are fixed 
C 768 29 881 0.574 512 512 24 in place with dental cement, a custom- 
K 576 29 483 0.412 128 256 This work designed 3D-printed headcap is added to 
M = - pha ie aa 1} 26)23,2637 4he implant to provide protective housing 
N 384 54 357 0.465 384 N/A 22,23,36,37 : : 
for the microelectrodes and electronics 
I 160 20 224 0.698 128 N/A 38 hades is fi : 
cl 160 12 7 0.240 128 N/A 20,29,39 (Fig. 1c,d). The headcap is fitted in a mod- 
G 128 7 114 0.448 128 N/A 20,29,39 ular manner. First, a conical base module 


10 x 10 arrangement). Each guiding tube accommodates bundles 
of 3-10 microwires of different length (Fig. la). Each bundle 
includes a single leading microelectrode with a conical tip; the 
remaining microwires have cut angle tips. We call these implants 
volumetric because they record from a volume of cortical (or 
subcortical) tissue (Supplementary Fig. 1). The microelectrodes 
are made of stainless steel microwires, 30-50 Um in diameter, 
with polyimide insulation that leaves the tip exposed. The guid- 
ing tubes are fixed in a 3D-printed plastic case, which also holds 
miniature screws for positioning the microelectrodes. The result- 
ing recording cubes are light and compact: a fully assembled unit 
weighs 11.6 g and has a surface area per channel of ~0.22 mm?. 
A total of 4-8 recording cubes can be implanted per monkey 
(Supplementary Fig. 2). 

During an implantation surgery, the guiding tube array is 
fixed in light contact with the cortical surface, without pen- 
etrating the brain. Several days later, the microwire bundles 
are advanced into the cortical tissue by rotating a set of micro- 
screws. Depending on the design, each microscrew turn advances 
microelectrodes housed in one (Supplementary Fig. 3) or several 
guiding tubes (Fig. 1a). In our previous work with fixed arrays, 
the microelectrodes were inserted in the cortex during surgery, 
and although these implants were less durable than our move- 
able arrays, histology performed on animals with these arrays 
showed little damage to cortical layers® (Supplementary Fig. 4). 
Using movable arrays, we have learned that penetration with sub- 
sets of microelectrodes reduces dimpling of cortical surface and 
copes with the ‘bed of nails effect’, which often hinders simulta- 
neous penetration by many microelectrodes in traditional arrays 
and silicon probes. In our implantation approach, each micro- 
electrode bundle is lowered gradually over the course of several 
days. This allows microelectrodes to be positioned more slowly 
and also much more deeply into the cortical tissue (i.e., layer VI). 


Figure 1 | Recording cubes and primate headcap. (a) Schematics 

and photographs of actual microwire cubes, showing the wire density 
and adjustment mechanisms. The third generation is a 10 x 10 array that 
is fitted with bundled microwires. (b) Photographs of final arrangements 
of connectors, showing the top of the headcap of a monkey implanted 
with second-generation cubes (left) and a similar view of the large-scale 
brain array module on a monkey implanted with third-generation cubes 
(center). Right, wireless module mounted on the headcap. (c) Layered 
schematic showing the modular headcap assembly, with the large-scale 
brain array connector organizer as the active module. (d) Schematic 
showing the modular headcap wireless assembly. Both headcaps 

are 3D printed. 
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is fixed on the top of the skull, which pro- 
tects the implants and their connectors. 
Depending on the application, several additional modules can 
be attached to this base to accommodate electronics. For example, 
a wireless-compatible module houses radio transceivers and their 
power source (Fig. 1d). Omnetics 36-pin connectors are employed 
to connect both external cables and the wireless headstages. 
For all monkeys except monkey O, the connectors were fixed 
with dental cement in between the cubes (Fig. 1b). For monkey 
O, implanted with 1,792 microwires, a special module organized 
all connectors (Fig. 1c). 


Multichannel recordings 

Monkeys M and N were the first animals to be implanted with 
four recording cubes containing 96 microwires each (totaling 384 
microwires per monkey). The cubes were placed bilaterally in the 
primary motor (M1) and primary somatosensory (S1) cortices, 
(Supplementary Fig. 2). Then, using a more compact revision, 
we implanted six cubes bilaterally in the arm and leg represen- 
tation areas of monkey K’s M1 and S1, totaling 576 channels. 
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Figure 2 | Large-scale activity recordings. (a) Representative mean 
waveforms for 1,874 neurons over a center-out task. Coloring indicates 
the array of origin: green, left-hemisphere primary motor cortex M1 
(LM1); orange, left-hemisphere primary somatosensory cortex S1 (LS1); 
blue, right-hemisphere M1 (RM1); yellow, right-hemisphere $1 (RS1). 
(b) Population raster plot depicting a 1-s window of neuronal spiking 
activity for a single recording session (565 neurons). The vertical color 
key identifies the recorded areas. A 128-channel Plexon system was 
used for each recorded area. (c) Unit isolation clustering using the first 
two principal components of a subset (50) of channels from monkey 0. 
Each color in the plots represents labeling of individual unit principal- 
component analysis clusters. Colored shading indicates the array of 
origin as ina. 


Monkey C was implanted with eight of these recording cubes, 
bilaterally in the supplementary motor area (SMA), M1, poste- 
rior parietal cortex (PPC), dorsal premotor cortex (PMd) and S1, 
totaling 768 channels (Supplementary Fig. 5). The latest revision 
allowed us to implant four 448-channel cubes, containing a total 
of 1,792 microelectrodes, in four cortical areas (448 microwires 
in each area, $1 and M1 in both hemispheres) of monkey O. 
One month after the original implantation surgery, extracellular 
neuronal recordings were obtained from these implants through 
four 128-channel recording systems (Plexon) (Fig. 2a,b), for a 
total of 512 channels recorded simultaneously. A total of 1,874 
single cortical units were isolated in monkey O at the end of 
four consecutive daily recording sessions (each day recording 
512 channels). 968 units were recorded in M1 (left hemisphere, 
403; right hemisphere, 565), and 906 units were recorded in S1 
(left hemisphere, 254; right hemisphere, 652) (Fig. 2b,c). 


Long-term recordings 

The longevity of our implants was assured through sterile surgery 
performed by an experienced neurosurgeon, regular (1-mm) spac- 
ing between implanted guiding cannulas, and slow insertion of 
microelectrodes into the brain. In this sample, monkeys I, Cl and 
G had older-generation fixed arrays, whereas monkeys M, N, C, K 
and O had the movable recording cubes. Altogether, this approach 
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has yielded high-quality neuronal recordings from eight rhesus 
monkeys. To our knowledge, our recordings®!7~*4 supersede the 
longevity reported with silicon probes”? or any other method thus 
far. This finding is further illustrated by a 4-year (2009-2013) 
sample of neuronal waveform traces, obtained from monkeys M 
and N (Fig. 3). At the time of writing, the recording duration in 
these animals has reached close to 5 years. Additionally, we have 
been recording from monkeys K and C (both females) for over 
29 months. Monkey O (female), implanted 
with the most advanced arrays, has been 


- - Monkey M 
5 al Vall aul ani antiemetic 5 FEV VER Ve ne recorded from for 13 months. We recorded 
S S ve a ye 8 ~~ Monkey 6 from monkeys I, Cl and G for 20, 12 and 7 
Sm a : 
ey gree ye a ~ = Monkey | months, respectively. Generally, the aver- 
RN te oh al ive AY fl anal Y a yn ee age yield (Fig. 3 and Table 1) for stable 
ee: = = Monkey Cl implants remained around 0.5-1.0 units 
s (over time) per microwire, a value that 
B oshe---7 7 %e i agrees with our previous results obtained 
$2 1 “ 
2 20; 1 ‘ 
g = Rs id Figure 3 | Recording longevity. Neuronal 
3 154 i recording yield over time measured by 
© "te ‘call quantifying the average number of single 
< 10} ne units recorded per connector (32 channels). 
All the data from the five monkeys used in these 
2008 2009 2014 : 
5 f f fj experiments, and from three other monkeys 
(G, I and Cl), were used. Four-year sample 
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s.d. These are taken from a single session on 
the corresponding date. 
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Figure 4 | Wireless brain-machine interface a 
(monkey C data). (a) Monkey with wireless 
system performing a task using pure 

brain control (Supplementary Video 1). 

(b) Performance metric increase in a single 
week of training for a single monkey. 

(c) Example average traces for the y coordinate 
of the cursor for trials with a target at 0° 
(north) of orientation, showing average 
constant movements despite variable postures 
during task interaction, as well as time 

to target. Peaks indicate reached target. 

(d) Peri-event time histograms centered on 
target onset for 212 units, grouped at each 
column with cursor control paradigms of 

basic hand control (HC) with joystick, brain 
control with joystick (BCWH) and sole brain 
control (BCWOH). The color key shows unit 
groupings per area. Each unit was normalized 
per condition using the z-score method. RSMA, d 
right-hemisphere supplementary motor area. 
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using adaptations of this methodology in 
rodents* and human patients*®?’, 


Neurons 


Large-scale data processing 

Processing large-scale data sets is a com- 
putationally demanding task. To handle 
real-time operations, we had to develop 
routines to process continuous streams of 
neural data and behavioral parameters. We 
also had to find a way to scale the number 
of neural channels. We solved the first issue by developing a real- 
time BMI suite (Supplementary Table 1)—a software package 
written in C++ for the Windows platform—that implemented 
all basic components needed to process large-scale activity and 
route it to control external devices (Supplementary Fig. 6). 
For the second issue of input scaling, we developed approaches for 
both tethered and wireless recordings. In our tethered recordings, 
which sample the full waveform of all channels, we developed 
networked spike acquisition software, which aggregated spikes 
acquired from multiple computers. In our wireless recordings, 
the system was designed to obtain waveform samples solely at the 
start of the session, when spike sorting parameters were gener- 
ated. This enabled considerable scaling owing to the low band- 
width used. 


Wireless recording system 

Our wireless recording system was designed to match typical 
BMI needs (Supplementary Table 2). This system consists of 
four components: digitizing headstages, a wireless transceiver, 
a wireless-to-wired bridge and client software (Supplementary 
Figs. 7-9). Four such headstages are attached to a transceiver 
module for a total of 128 channels per transceiver unit (31.25 kHz 
sampling per channel, 1.3 kHz sampling of spike time onset). 
Multiple 128-channel modules can be stacked together to scale up 
the final system. Our system's novel feature, bidirectional commu- 
nication, enables spike sorting to be performed on the transceiver 
itself, thereby saving both power and wireless bandwidth. This is 
accomplished through key elements of the transceiver: the radio 
and the digital signal processor (DSP). The DSP performs signal 
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conditioning, spike sorting and radio control to match the radio 
bandwidth with the 48-Mbps aggregate rate of data acquisition. 
Each transceiver is powered by a 3.7-V, 2,000-mA h lithium-ion 
cell and consumes approximately 2 mW per channel, or ~264 mW 
per 128-channel unit, allowing continuous operation for over 
30 h. The wireless-to-wired bridge takes incoming radio packets 
and bridges them with an Ethernet interface. Overall, this signal 
chain is optimized in both temporal and power domains, allow- 
ing for the increased number of channels. Additionally, because 
of the low bandwidth requirements, each transceiver uses a 
small slice of the industrial, scientific and medical (ISM) radio 
band. This allows substantial scaling on the number of recording 
channels. We benchmarked our wireless system performance 
using four transmitters, or 512 channels simultaneously, by 
measuring packet-bit drop errors (PER/BER) versus transmis- 
sion range. Overall, the system performed within its optimal 
range (3 m) with all four modules (Supplementary Fig. 10a). 
We then used this 512-channel system to record brain activity 
while monkey C reached for grapes. Out of the 512 wirelessly 
recorded channels, we isolated a total of 494 units (103 M1 units, 
102 S1 units, 98 SMA units and 191 PPC units). Cortical units 
exhibited typical firing-rate modulations as the animals produced 
arm-reaching movements (Supplementary Fig. 10b). For other 
experimental validation, we used two transceivers per monkey 
(i.e., 256 channels). 


Wireless brain-machine interface 
The real-time operation of our wireless system in a typical BMI 
setup was tested in monkey M (128 channels in one group of 


& © 2014 Nature America, Inc. All rights reserved. 


Figure 5 | Recordings of freely behaving a 
monkeys. (a) Screenshots from videos 

used for coding of behaviors. Two frames 

each for three sample behaviors are shown. 

See Supplementary Video 3. (b) First three 
principal components of neural activity for 

six different behaviors in monkey K. 
Principal-component analysis data were used 
for support vector machine classification 

(see Supplementary Table 3). (c) Peri-event 
time histograms for behaviors from monkey K, 
calculated from 107 M1 neurons and centered 
around behavior onset. The spectrogram 

legend is on the bottom right. Each unit 

was normalized for the entire session 

(all behaviors shown) using the z-score method. 


Walk 


experiments, and 256 in the other), mon- 
key C (256 and then 512 channels) and 
monkey K (256 channels). These monkeys 
performed several BMI and behavioral 
tasks without any restraint. While inside 
a large Plexiglas enclosure (Fig. 4a), mon- 
keys C and M learned to use their brain 
activity to move a computer cursor to 
execute a classical center-out task (Online 
Methods). Initially monkey C used a joy- 
stick to perform the task manually (the 
HC, or ‘hand control, condition). Monkey 
C was then required to perform through 
a BMI, with a Wiener filter used as the decoder?®. During the 
‘brain control with hand’ condition (BCWH), the joystick was still 
accessible to the monkey, although it was disconnected from the 
cursor control. During ‘brain control without hand’ (BCWOH), 
we removed the joystick from the setup, forcing the monkey to 
perform the task using its cortical activity alone (Supplementary 
Video 1). A total of 212 cortical units recorded from M1 (84 and 
26 units in the left and right hemispheres, respectively), SMA 
(27 and 24 units) and left hemisphere S1 (51 units) were used. 
Performance improved (reaching >80% correct trials) within 7 d 
(Fig. 4b). Population peri-event time histograms (PETHs) for 
all recorded neurons, centered on target onset, show neuronal 
adaptation to each task component (Fig. 4). Cortical ensembles 
exhibited similar patterns of modulation in BCWH and HC, peri- 
ods during which the animal moved the joystick. Different neuro- 
nal modulation patterns were observed in BCWOH, likely owing 
to decreased body movements. In particular, many units were 
prominently inhibited before target onset. Additionally, excitatory 
neuronal modulations occurred during reaching for all modes of 
operation but had lower amplitude and longer duration during 
BCWOH. We then replicated this task using a powered wheelchair 
as an end effector (Supplementary Video 2). 

To address the relevance of the number of recording chan- 
nels for BMI decoding, we conducted an offline analysis known 
as neuron-dropping curves’ by randomly removing units from 
the training set of the decoder (unscented Kalman filter?®). The 
performance of the decoder was measured as the correlation 
coefficient (r) between decoded and measured variable. We show 
neuron-dropping curves for two monkeys (O and C) performing 
the center-out task and for two monkeys (M and N) perform- 
ing a bipedal locomotion task on a treadmill (Supplementary 
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Fig. 11). For all tasks and monkeys, model performance increased 
linearly as a function of the logarithm of the total number 
of neurons!®, 


Wireless recordings in freely moving monkeys 

We also explored the capacity of our wireless system to record 
from freely moving monkeys in an exploratory foraging task. 
Monkeys K and C were allowed to freely move about an experi- 
mental room (Supplementary Video 3) while picking grapes 
from locations on the room floor and on suspended planks. 
Recordings included a total of 247 units from monkey C (193 in 
M1; 64 in S1) and 156 units from monkey K (107 in M1; 49 in 
$1). Our wireless system recorded reliably inside the 3-m range of 
the recording equipment (Fig. 5a and Supplementary Video 2). 
Population PETHs aligned on the onset of different behaviors 
showed distinct neuronal modulation patterns (Fig. 5c). We then 
used several clustering methods (k-means, expectation maximiza- 
tion and support vector machines (SVMs; linear kernel and radial 
basis function)) to discretely classify several observed behaviors. 
All four clustering methods employed yielded good performance 
(Supplementary Table 3) in correctly identifying six selected 
behaviors (Fig. 5b) on the basis of the combined electrical activity 
of cortical ensembles. 

We did further experimental validation with monkey K while it 
performed an unrestrained locomotor task. The monkey walked 
both bipedally and quadrupedally on a treadmill while we wire- 
lessly recorded from 176 units in M1 (left hemisphere, 94; right 
hemisphere, 82). Population PETH analysis of left M1 neurons 
during the swing phase of the right ankle revealed an inversion of 
firing patterns in a subpopulation of these cells (Supplementary 
Fig. 12). 


NATURE METHODS | ADVANCE ONLINE PUBLICATION | 5 


236 


& © 2014 Nature America, Inc. All rights reserved. 


237 


| ARTICLES 


Next-generation implants 

Continuous improvement of our arrays is necessary to further 
explore the relevance of large-scale recordings in BMIs and neuro- 
physiology. We have introduced a manufacturing modification 
(Supplementary Fig. 13a,b) that allows us to increase the number 
of microelectrodes per individual cannula in our recording cubes 
to up to 30 microwires and a new cap that accommodates close to 
10,000 microwires (Supplementary Fig. 13c,d). 


DISCUSSION 

To meet future experimental demands, chronic multielectrode 
implants should fulfill the following requirements: (i) produce 
minimal damage to neural tissue, (ii) maximize the number of 
simultaneously recorded neurons, (iii) sample from multiple 
brain areas, and (iv) maintain good recording quality for several 
years. Over the last two decades, our laboratory has progressively 
improved the performance of multichannel recording systems, 
moving toward these goals through a series of technological 
developments and experimental tests in rodents, primates and 
humans (via intraoperative recordings)?®3>!, Here we described 
a paradigm that allows wireless large-scale brain recordings to be 
carried out in freely roaming primates. 

The first step of this paradigm involved the introduction of 
volumetric recording probes, which can be used to sample from 
hundreds to thousands of neurons, distributed across multiple 
cortical areas per monkey. Using this integrated methodology, 
we increased the number of cortical units recorded simultane- 
ously per monkey in a single recording session (~500 units)*4, the 
total number of units isolated from a single animal (~2,000 units) 
and the longevity of chronic recordings (close to 5 years). To our 
knowledge, none of these milestones has been reached by another 
integrated system for chronic recordings in behaving monkeys. 
For comparison, the Utah silicon probe, commonly used for BMI 
studies and other neurophysiological experiments in primates, 
provides fewer recording channels (~100), penetrates only the 
superficial cortical layers, does not allow adjustment of the micro- 
electrode’s position after the implantation surgery, and cannot 
yield long-term cortical recordings beyond a few months?>*. 

The large increase in the number of neurons recorded has 
not compromised the stability and reliability of our implants. 
Although we have not performed histology for the most recent 
implants, we have previously shown that implanted microwires 
produce minimal tissue damage’. This is further demonstrated 
by the fact that high-quality neuronal data could be recorded from 
two monkeys for nearly 5 years and in another two monkeys for 
nearly 3 years. Although neuronal yield decreases between the 
first and third after the surgery, we showed that recordings stabi- 
lize at an average of 0.5-1.0 units per channel for many months 
or even years in most monkeys. A similar yield has been observed 
over the past 20 years in our laboratory in the recordings from 
rodents‘, owl monkeys’ and human subjects?®’, 

Moreover, the performance of BMI decoders increases line- 
arly with the logarithm of the cortical neuronal sample recorded 
simultaneously’. Therefore, increasing the number and long- 
term stability of multielectrode recordings is imperative for 
the development of clinically relevant neuroprostheses driven 
by BMIs. Indeed, we have recently employed the technology 
described here to implement the first bimanual BMI, which used 
approximately 500 units to control a pair of virtual arm actuators”4. 
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We envision that by doubling or tripling the sample of neurons 
recorded simultaneously, even richer BMI applications, such as 
a robotic exoskeleton aimed at restoring full-body mobility, may 
become viable. 

Previously, our group!* and other laboratories!#!>!5 have 
described multichannel wireless recording technologies. Although 
most of these systems improved the design, size, energy efficiency 
and depth of implants of the components (Supplementary Table 4), 
our current approach introduces several notable innovations. 
The key innovation is its ability to maintain bidirectional wireless 
communication, an essential property missing in the literature. 
Bidirectional wireless communication is a crucial component 
for scaling up the number of channels recorded simultaneously, 
as it allows spike sorting to be performed on the implant and, 
therefore, reduces the amount of information transferred wire- 
lessly. Using this approach, we recorded up to 512 channels of 
neuronal data in the present study. Overall, considering only the 
ISM band, we estimate that in the future, up to 20 128-channel 
transceivers could be used, each occupying 2-MHz increments, 
for a total of 2,560 recording channels. Because two neurons could 
be isolated per channel, this technology could yield up to 5,120 
neurons recorded simultaneously in a single monkey. However, 
a few bottlenecks, including size and power consumption, must 
be addressed before this benchmark can be reached. 

Additionally, removing the need for hardware tethering is 
extremely advantageous for primate experiments, particularly 
when there is growing evidence that the context in which animals 
perform a given task dramatically influences the physiological 
properties of neural circuits*®. For example, studies focused on 
how cortical circuits underlie natural behaviors such as tool uti- 
lization, social primate interactions and complex movements*475 
are likely to benefit from our new recording paradigm. 

Use of a novel wireless large-scale recording approach as 
described here may prove ideal for studies of large-scale brain 
activity under a variety of normal and pathological conditions, 
and it may serve as both the neurophysiological and technological 
backbone for the development of future generations of BMIs and 
clinical neuroprostheses. 


METHODS 
Methods and any associated references are available in the online 
version of the paper. 


Note: Any Supplementary Information and Source Data files are available in the 
online version of the paper. 
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ONLINE METHODS 

Animals. All animal procedures were performed in accordance 
with the US National Research Council’s Guide for the Care 
and Use of Laboratory Animals and were approved by the Duke 
University Institutional Animal Care and Use Committee. Five 
adult rhesus macaque monkeys (Macaca mulatta; one male, four 
females) participated in the set of experiments (monkeys M, N, 
C, Kand O). Additionally, data from three monkeys (Cl, G and |; 
two males, one female) were used in the longevity analyses. 
Histology data from one additional monkey (T, one female) were 
used in this report. 


Surgery. Adult rhesus macaques were initially anesthetized 
with ketamine and then intubated in the trachea. They were 
then placed in a stereotaxic apparatus and ventilated. General 
anesthesia was maintained with isoflurane throughout the sur- 
gery. A series of craniotomies were performed over the stere- 
otaxically determined locations of cortical areas of interest. The 
dura mater was removed, and sulci landmarks were inspected 
to confirm the location of cortical areas. Microwire cubes were 
brought in light contact with the pia mater and fixed to the bone 
with bone cement. Titanium screws were inserted in the skull to 
anchor the cement. Gelfoam coated in saline was inserted into the 
pia to sustain the surrounding tissue. After all recording cubes 
were implanted, a protective headcap was affixed to the screws 
and cemented to the skull. The cap encased the cubes and their 
connectors and provided housing for wireless components. The 
dura, dense connective tissue and, later, skull tissue grow back to 
capture and seal each of the microelectrode array/bundle pen- 
etration point. We have observed such tissue restoration in all 
implanted monkeys examined post-mortem. Because of these 
processes, the risk of infection growth around the microelectrode 
implants decreases substantially over the first few weeks after the 
implantation surgery. 


Implantation. Once the animals recovered from the surgery 
(usually 1 week post operation), we began the insertion of the 
microelectrodes. The potential for infection is vastly reduced dur- 
ing the first post-surgery week by using antibiotic gel to cover 
the entire implant. Following the surgery and array implanta- 
tion, microwire bundles were inserted slowly into the brain. Each 
microelectrode bundle was moved gradually over the course of 
several days. This was achieved by turning microscrews in the 
array assembly, which lowered subsets of microwires independ- 
ently. Each quarter turn of the microscrew lowers the bundle by 
53 um. As expected, microelectrodes did evoke a tissue reaction 
over the first few weeks after implantation. This tissue reaction, 
which included a local immunological response, was contained 
and did not lead to any significant tissue damage outside the 
microelectrode penetration rack. 


Implants. Because the cortices of rhesus monkeys are convoluted, 
the implants were developed to be suited for both relatively shallow 
penetrations in cortical gyri and deeper penetrations in the sulci. 
As such, our recording cubes contain guiding tubes loaded sepa- 
rately for gyri and for sulci. For gyral penetrations, guiding tubes 
are fitted with 2-5 microelectrodes staggered at 0.3-0.5 mm. For 
sulcal penetrations, 5-10 microelectrodes are staggered at 0.5-1.0 
mm. Using this approach, the density of microelectrodes within 
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a cortical volume can be adjusted, depending on the depth of the 
targeted cortical area. Monkeys M and N were implanted with 
four 96-microwire recording cubes constructed of stainless steel 
304 (30-50 um in diameter). Each hemisphere received two cubes: 
one in the upper-limb representation area and one in the lower- 
limb representation area of sensorimotor cortex. Monkeys C and 
K were implanted with 128-microwire cubes (stainless steel 304, 
30-50 um in diameter). Monkey K received two cubes in M1 and 
one array in S1 for each hemisphere. Monkey C received two cubes 
in M1, one in S1 and another in SMA. Monkey O was implanted 
with four 448-channel cubes (stainless steel 304, 30-50 Um in 
diameter), one in M1 and one in S1 for each hemisphere. Data 
from three previous monkeys (monkeys I, Cl and G) were used 
for longevity analysis. Monkeys I, Cl and G were implanted with 
96-microwire arrays. Histology data from one additional monkey 
(T) was used in this report. Monkey T was implanted with three 
32-microwire (stainless steel 304, 50 um in diameter) fixed arrays. 


Neuronal isolation. After electrode insertion, we examined each 
microwire for units and made adjustments to the cortical depth of 
the wire on the basis of these results. After microwire positioning, 
we performed spike sorting for use in our experiments, monitor- 
ing the units on an almost daily basis (Supplementary Videos 4 
and 5). Units were sorted using standard template-matching para- 
digms*? available in our Plexon and wireless recording systems, 
which are standard and well used across the literature!®. To assess 
whether recorded units were single or multiple, we estimated 
refractory period using interspike intervals. Units that exhibited 
refractory periods greater than 1.6 ms (ref. 9) were considered 
single units. Population plots included data from both single units 
and multiunits, so recorded units in these plots will be referred to 
simply as ‘neural units’ or ‘units. 


Histology. Histology for microelectrode tracts was performed in 
one animal (monkey T). The animal was deeply anesthetized with 
pentobarbital and transcardially perfused with 4% paraformalde- 
hyde. Sections of representative areas of interest of the brain were 
selected, prepared and stained with cresyl violet (Nissl method) 
in order to identify electrode track locations. Slices were prepared 
using a HMH 505 E cryostat (Microm) with 50-11m thickness. 


BMI suite. Our BMI suite is written in C++ and DirectX 9.0. It 
uses Lua bindings for configuration and scripting, and it interfaces 
directly with Plexon using the C++ Plexon API. The BMI suite 
also interfaces with our custom-designed wireless system using a 
remote procedural call (RPC) service written in the wireless client 
that transmits the firing rate of all neurons at every request (the 
BMI RPC calls at 100 Hz). Interfacing with other data streams is 
usually achieved through a UDP socket or, less often, a simple 
serial interface. Other signal input sources may be added in the 
future, using all existing algorithmic methods, without significant 
modification of the existing infrastructure because of the modu- 
larity of the design. Decoding algorithms operate using a common 
interface with the BMI software suite, allowing new routines to be 
implemented in a modular way. Algorithmic outputs can be com- 
bined flexibly at the user’s behest using the software's GUI and then 
passed to the experiment manager module. The experiment sub- 
routine implements the behavioral task and handles stimuli, feed- 
back and experimental procedure logic. The experiment manager 
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module uses a standardized interface, which allows different 
experiments to be switched in and out without affecting the rest 
of the software suite. Finally, experimental data are recorded 
through a unified threaded logging system, which automatically 
records all pertinent experimental variables. 


Wireless transceiver. An Analog Devices Blackfin BF532 DSP 
performs signal conditioning, sorting and radio control on the 
transceiver and is written in assembly language. The Blackfin 
processor uses sum absolute accumulate (SAA) instructions, 
typically used in MPEG video compression, in order to imple- 
ment L1 norm template sorting. The analog-to-digital converters 
(ADCs) output four samples every microsecond, and processing 
each proceeds in lockstep. First they are dithered to 14 bits viaa 
low-pass prefilter, high passed at 250 Hz via an integrator, and 
then gained further from 0-128 (7.8 fixed point) with a 1-bit 
stochastic automatic gain control (AGC) block. This is followed by 
a 1-bit-normalized least mean squares (LMS) adaptive noise can- 
cellation wherein the last 14 circular-buffered sampled channels 
are used to linearly predict the present channel. Then this predic- 
tion is subtracted, and the error is saturated to 1 bit (plus sign) and 
used to update the weights. Only one weight is updated at a time, 
but as noise is largely uncorrelated with this rate (2.232 kHz), 
it does not affect the rejection ratio of >40 dB. Noise-canceled 
samples are fed to eight poles of host-configurable IIR biquads, 
implemented as Direct Form-I with coefficient prescaling to pre- 
vent fixed-point saturation. The biquads are implemented with 
ping-pong register buffering and share taps between subsequent 
stages for maximum efficiency. Importantly, these biquads can be 
configured as oscillators, which has proved useful when debug- 
ging radio function. 

On each headstage, a 32-channel amplifying and multiplexing 
chip (Intan RHA2132) feeds a 12-bit ADC (ADCS7476; 31.25 kHz 
sampling on each channel). Digital signals are isolated from the 
analog inputs with LDO regulators. 

The Blackfin processor has a special SAA instruction that is 
intended to speed MPEG video compression by measuring the 
block difference between frames; here we use it to rapidly imple- 
ment L1 norm template-based sorting. As the SAA instruction 
operates on a four-sample 32-bit word, samples from each of 
the four headstages are shortened to 8 bits and merged into one 
word and placed into a circular buffer; the sum of absolute dif- 
ferences of 16-sample stretches of waveform are then computed 
and compared to a threshold. If they are below the threshold, a 
spike is marked as detected via a sticky bit in a radio FIFO. Two 
such templates are implemented per channel (templates A and B) 
for a total of 256 templates per unit. Note that there is no other 
threshold operation; template matching for both units per channel 
is computed for every incoming sample. 

Wireless transmission uses a Nordic Semiconductor radio 
(nRE24L01+; 2.4GHz; 2 Mbps unidirectional bandwidth; 1.333 Mbps 
with overhead and direction switching). Signal conditioning 
occurs in synchrony to radio control, as there are no threads 
on the system; radio control state is implicitly encoded in the 
program counter. Radio data are sent as cyclic redundancy check 
(CRC)-protected 32-byte packets at a rate of 5,208.3 packets/s; the 
radio control code fills these packets with six samples from each 
of four continuously transmitted channels, plus 8 bytes of tem- 
plate match. This equates to all 256 bits of template match being 
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transmitted at 1.3 kHz. However, this packet leaves no space for 
synchronization or acknowledgment communication. Accordingly, 
given that L1 template matching is unable to accurately detect 
two simultaneous action potentials (templates A and B), 
we drop a few of these unlikely events via a lookup table com- 
pression, freeing up one extra bit per byte in the template-match 
fields. Four such bits are used to synchronize the transceiver and 
bridge, as every 16 outgoing packets the transceiver transitions to 
receive a command packet. The other 4 bits are used as an echo 
field to verify command-packet reception. This radio protocol, 
including state transitions, utilizes 99.6% of available bandwidth 
with the Nordic radio chip. The whole transceiver software, which 
is written in assembly, fully processes a sample in less than 80 
instructions and consumes only 12.6 kB of L1 cache. 


Wireless bridge. Rather than using polarized antennas, the bridge 
uses three identical Nordic radios with orthogonally arranged 
antennas; radio packets are accepted from any source provided 
they match the CRC. Successful reception of a data packet tran- 
sitions the receiving radio to transmit mode and begins bidirec- 
tional communication. There are four pairs of addresses and data, 
which allow all elements of the signal conditioning path to be 
varied on the transceiver, for example, to set template, vary band- 
pass filter, disable AGC or set which channels are being continu- 
ously transmitted. The bridge also features a full protocol stack, 
including IP, ICMP, DHCP, TCP, HTTP and SLP, the last of which 
allows software discovery of bridges. Finally, the bridge features 
audio output, transceiver programming circuitry, and a power- 
over-Ethernet (PoE) module, so it can be remotely powered, for 
example, within an animal room. 


Wireless client. The sorting client is written in C/C++ using the 
GTK2 GUI toolkit with OpenGL and HLSL for graphics, presently 
on Debian GNU/Linux. The software allows further GUI control 
of all elements of the signal-conditioning path, as outlined above. 
The client software allows for direct visualization of waveforms 
from four selected channels and all spike channels, while enabling 
spike sorting, data collection, signal chaining, and interfacing 
with multiple bridges and transceivers. For spike sorting, two 
units can be sorted per channel, for a maximum of 256 units 
per transceiver. Unit isolation and sorting is performed with an 
algorithm based on principal component analysis (PCA): the user 
specifies a threshold to obtain waveform snippets, which are then 
represented by the first two principal components on a 2D plane. 
The user then marks the clusters of dots that represent neuronal 
waveforms in the PCA space. This software will be available upon 
request, and will be available in the near future as open source 
(as well as the wireless hardware) under GPL v3. 


Unrestrained center-out task. The task requires placing a compu- 
ter cursor in the middle of the screen and, upon the appearance of 
a target, placing the cursor over the target for a juice reward. Each 
trial commenced with a circular target appearing in the center of 
the screen. The monkeys held the cursor within this target for a 
delay randomly drawn from a uniform distribution (from 500 to 
2,000 ms). After this delay, the central target disappeared and a cir- 
cular target appeared on the screen at a distance of 7 cm from the 
center, in one of eight directions. The spacing between the objects 
was 0.1 cm. Before this task was run inside the Plexiglas enclosure, 
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animals were trained while seated in a primate chair. The enclo- 
sure was large enough (1 m x 1 m x 1 m) to allow the monkeys 
freedom of movement. The monkeys were free to perform the 
task at any time and accommodated to the enclosure quickly. 
In only 3 d, they started performing the manual task. Monkeys’ 
behavior was captured throughout each session by a video camera. 
For the long-term tasks, data collection was done by running an 
Ethernet PoE cable from a laboratory computer to the home cage 
and replacing the wireless system's batteries every 24 h. 


Locomotion task. Three monkeys were involved in locomotion 
studies (M, N and K). They were trained to walk bipedally on a 
custom-modified treadmill. A standard neck restraint was sus- 
pended from a linear actuator, which allowed bipedal support to 
be adjusted gradually while the monkey achieved bipedal posture 
and weight bearing. A juicer was placed in front of the macaque 
and could be raised or lowered depending on the macaque’s height 
and the locomotor task (bipedal or quadrupedal walking). A video 
monitor mounted in front of the monkey displayed visual feed- 
back for tasks, training and enrichment. Only monkey K used 
wireless recordings and walked quadrupedally. Monkey M and N 
walked only bipedally and were recorded with our Plexon system, 
and their results were used in the dropping-curve analysis. 


Powered wheelchair. A conventional motorized wheeled chair 
was modified for computer-controlled movements. The human 
chair was dismounted, and the wheeled base was interfaced with a 
Robotics motor controller (RoboteQ Model VDC2450) based on 
a dual-channel microprocessor. Computer control was established 
using a 900-MHz serial pipe (APC, Wireless RS232) to transmit 
serial commands from the computer to the controller at a rate of 
9,600 baud. 


Wheelchair task. Using the protocol for the center-out task, we 
applied data from the cursor in both BCWH and BCWOH para- 
digms as input into a differential-drive control law that was used 
to drive the wheelchair through the room (3 m x 3 m x 4 m). 
An additional cursor was shown on the screen, which represented 
the wheelchair following the monkey’s cursor position. To provide 
this additional feedback, the wheelchair’s coordinates were video 
tracked and transformed onto the screen. Additional rewards 
were administered when the wheelchair cursor hit the target. 
The algorithm of wheelchair control set the linear and angular 
velocities of the wheelchair as functions of the monkey cursor 
position, and programmable boundaries kept the wheelchair from 
striking the walls by setting linear velocity to 0 when outside the 
boundary of the working area. A sample video of the task is shown 
in Supplementary Video 2. 
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Free roaming experiments. Monkey K was videotaped, and 256 
cortical units were simultaneously recorded in this animal while it 
roamed freely in a room (3 m x 3m x 4m). Offline, the monkey’s 
motions were hand labeled into 15 categories of behavior. 
Neuronal data were aligned with video recording; neurons that 
did not fire over 500 times in 1 h were eliminated (leaving 107 
neurons), and PETH analysis was performed with 2-s windows 
and 100-ms bins. Population PETH data were calculated and 
analyzed as spectrograms. For discrete predictions, six behaviors 
were selected on the basis of their saliency and ease of identifica- 
tion, as well as the amount of data available to them (>30 events 
per behavior). 


Kinematics tracking. An in-house—developed tracking system 
was used to track the three-dimensional position of the ankle, 
knee, hip, wrist, shoulder and head of monkeys, as well as the 
surface of the limbs. Both limbs were tracked using two units of 
this system. 


Decoding algorithms. Historically, BMI research has used Weiner 
filters for online predictions of movement. Recently in our labora- 
tory, we developed a more advanced algorithm for predicting limb 
movements, called the unscented Kalman filter’. This algorithm 
was used for the extraction of lower-limb kinematic parameters 
from cortical activity, as well as cursor parameters from joystick 
tasks. The Weiner filter was also applied when indicated. For dis- 
crete classification, we used Matlab implementations of EM, SVM 
and k means. 


Analysis. All data analyses were performed on Matlab. No inves- 
tigator blinding, nor animal randomization, was performed. 
PETHs were calculated taking a 1-s time window centered on 
target onset in the joystick tasks, or swing onset for the left leg 
in the locomotor tasks, and averaging binned firing rates over 
all trials for each neuron. Neuron activity was normalized per 
neuron, per condition. Analysis of recordings over months and 
years of experimentation was performed by script, reading all 
experimental files and counting average number of units recorded 
per 32-channel array. For behavioral clustering, we used imple- 
mentations of PCA, EM, SVM and k means available in the Matlab 
statistical toolbox. Analysis of neuron scaling performance was 
done by using all neurons, then randomly removing subsets and 
then cross-validating the process over several iterations. 


40. Lewicki, M.S. A review of methods for spike sorting: the detection and 
classification of neural action potentials. Network 9, R53-R78 (1998). 


doi:10.1038/nmeth.2936 


241 


frontiers in 


COMPUTATIONAL NEUROSCIENCE 


ORIGINAL RESEARCH ARTICLE 
published: 19 August 2014 
doi: 10.3389/fncom.2014.00091 


Signal-independent timescale analysis (SITA) and its 
application for neural coding during reaching and walking 


Miriam Zacksenhouse'*, Mikhail A. Lebedev? and Miguel A. L. Nicolelis? 


' Brain-Computer Interfaces for Rehabilitation Laboratory, Department of Mechanical Engineering, Technion - IIT, Haifa, Israel 
? Department of Neurobiology, Center for Neuro-Engineering, Duke University, Durham, NC, USA 


Edited by: 
Mayank R. Mehta, UCLA, USA 


Reviewed by: 

Alexander G. Dimitrov, Washington 
State University Vancouver, USA 
James McFarland, University of 
Maryland, USA 


*Correspondence: 

Miriam Zacksenhouse, 
Brain-Computer Interfaces for 
Rehabilitation Laboratory, 
Department of Mechanical 
Engineering, Technion - IIT, Haifa 
32000, Israel 

e-mail: mermz@tx.technion.ac. il 


What are the relevant timescales of neural encoding in the brain? This question is 
commonly investigated with respect to well-defined stimuli or actions. However, neurons 
often encode multiple signals, including hidden or internal, which are not experimentally 
controlled, and thus excluded from such analysis. Here we consider all rate modulations 
as the signal, and define the rate-modulations signal-to-noise ratio (RM-SNA) as the 
ratio between the variance of the rate and the variance of the neuronal noise. As the 
bin-width increases, RM-SNA increases while the update rate decreases. This tradeoff 
is captured by the ratio of RM-SNA to bin-width, and its variations with the bin-width 
reveal the timescales of neural activity. Theoretical analysis and simulations elucidate 
how the interactions between the recovery properties of the unit and the spectral 
content of the encoded signals shape this ratio and determine the timescales of neural 
coding. The resulting signal-independent timescale analysis (SITA) is applied to investigate 
timescales of neural activity recorded from the motor cortex of monkeys during: (i) 
reaching experiments with Brain-Machine Interface (BMI), and (ii) locomotion experiments 
at different speeds. Interestingly, the timescales during BMI experiments did not change 
significantly with the control mode or training. During locomotion, the analysis identified 
units whose timescale varied consistently with the experimentally controlled speed of 
walking, though the specific timescale reflected also the recovery properties of the unit. 
Thus, the proposed method, SITA, characterizes the timescales of neural encoding and 
how they are affected by the motor task, while accounting for all rate modulations. 


Keywords: timescales, rate modulation, doubly stochastic Poisson processes, rate coding, brain-machine interface, 


signal to noise ratio 


INTRODUCTION 

The timescales of neural encoding is important for understand- 
ing neural processing and for successfully interpreting recorded 
neural activity (Shadlen and Newsome, 1994; de Ruyter van 
Steveninck et al., 1997; Borst and Theunissen, 1999; Jacobs et al., 
2009). In the context of rate-coding, we focus on characteriz- 
ing those timescales, independent of which signals modulate the 
rate. Those may include: (i) encoded signals under experimen- 
tal control, (ii) encoded task-relevant signals that are not under 
experimental control and thus may vary from trial to trial, and 
(iii) task-irrelevant signals. The last two groups are referred to 
as “hidden” signals, and we are especially interested in the sec- 
ond group of hidden task-relevant signals. Those may include, 
for example, the estimated state or estimation error during reach- 
ing movements (Desmurget and Grafton, 2000; Wolpert and 
Ghahramani, 2000; Krigolson and Holroyd, 2006; Shadmehr and 
Krakauer, 2008). 

The potential contribution of hidden signals to trial-to-trial 
variability in neuronal responses has been noted in the con- 
text of different tasks, including skilled reaching movements 
(Churchland et al., 2006a,b; Mandelblat-Cerf et al., 2009) motor 
adaptation (Mandelblat-Cerf et al., 2009) and decision making 


(Churchland et al., 2011). During skilled reaching movements, 
trial-to-trial variations in preparatory neural activity were pre- 
dictive of variations in reach, and were interpreted to reflect 
variations in movement planning (Churchland et al., 2006a). 
During novel visuo-motor tasks, trial-to-trial variability in spik- 
ing activity increased in early adaptation stages before return- 
ing to initial levels at the end of learning, and was interpreted 
to reflect enhanced exploration or adjustments of the internal 
models (Mandelblat-Cerf et al., 2009). During decision mak- 
ing, trial-to-trial variations in the underlying rate were shown 
to increase until the decision was made, in agreement with the 
hypothesis that they encode trial-specific evidence accumulation 
(Churchland et al., 2011). 

Ideally we should quantify and analyze the contributions of the 
first two groups of signals, i.e., only those that are task-relevant, 
whether under experimental control or hidden. However, this 
is hampered by the inability to control or measure hidden 
task-relevant signals. Current methods for analyzing neuronal 
timescales, which have focused mainly on sensory neurons, are 
based on averaging the response over repeated trials with the 
same stimuli (Borst and Theunissen, 1999; Warzech and Egelhaaf, 
1999; Butts et al., 2007), or computing the mutual information 
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between known stimuli and their reconstruction (Rieke et al., 
1997; Dimitrov and Miller, 2000). In either case, the analysis 
accounts only for signals that are under experimental control, ie., 
the first group of signals. 

Neurons in the motor system are known to encode a large 
number of signals (Georgopoulos, 2000; Johnson et al., 2001; 
Scott, 2003), and are also expected to encode hidden signals, such 
as the estimated state and errors (Desmurget and Grafton, 2000; 
Wolpert and Ghahramani, 2000; Krigolson and Holroyd, 2006; 
Shadmehr and Krakauer, 2008). Timescale analysis that is based 
on synchronized averaging would miss the contribution of those 
signals that are not under experimental control, including hidden 
task-relevant signals. This is illustrated in Figure 1A (detailed in 
Section Rate-modulations Signal-to-noise ratio of DSPP), where 
the rate is assumed to encode both a movement related sig- 
nal and a hidden signal: synchronized averaging captures the 
rate-encoded movement but not the rate-encoded hidden sig- 
nal. Instead, we focus on analyzing the timescales associated with 
the underlying rate modulations, independent of which signals 
modulate the rate. The proposed method considers asynchronous 
sequences of random reaching movements (Figure 1B, upper 
panel) so the resulting rate (Figure 1B, lower panel) is a stationary 
process whose variance (dashed black) captures the variance of 
both of the encoded signals (dashed black lines, Figure 1A upper 
panel and Figure 1B lower panel, respectively). Thus, in contrast 


with standard averaging methods, the proposed method captures 
also the contribution of hidden signals. Admittedly, the hidden 
signals may include not only task-relevant (group 2) but also task- 
irrelevant (group 3) hidden signals. Hence, the method is best 
used in comparing the timescales across different task conditions. 
Here we demonstrate the power of this method by compar- 
ing the timescales in neural activity during different phases of 
experiments with brain-machine interfaces (BMIs) and during 
locomotion at different speeds. 

Timescales are assessed under the sole assumption that spike 
trains are realizations of dead-time modified doubly stochastic 
Poisson processes (DSPP) (Snyder, 1975; Cox and Isham, 1980; 
Johnson, 1996; Gabbiani and Koch, 1998). DSPPs are the simplest 
point processes that can describe rate modulations by stochas- 
tic signals. This model is pertinent for describing spike trains 
recorded during reaching and walking when the rate might be 
modulated by a number of biologically relevant stochastic signals. 
These signals may include not only measurable variables, such 
as the velocity of movement, but also internal or hidden signals 
(e.g., internal state estimations; Wolpert and Ghahramani, 2000; 
Shadmehr and Krakauer, 2008), which cannot be measured or 
controlled directly (Zacksenhouse et al., 2007). The effect of dead- 
time is explicitly investigated to account for absolute refractory 
period. The effects of other deviations from the DSPP assumption 
are demonstrated via simulations. 
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FIGURE 1 | Illustration of the averaging approach (A) vs. the 
proposed approach (B) when the rate encodes both the movement 
and a hidden signal. Averaging methods are based on repeating the 
same movement (A, upper panel) and synchronizing the resulting neural 
activity and hence the rate (A, lower panel). Synchronized average (solid 
black) is considered the signal, while the variance of the hidden signal 
(dashed black) is considered noise. The proposed method considers 
asynchronous sequences of random reaching movements (B, upper 
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panel) so the resulting rate (B, lower panel) is a stationary process 
whose variance (dashed black) captures the variance of both of the 
encoded signals [dashed black lines, (A) upper panel and (B) lower 
panel, respectively]. Each movement has a minimum jerk profile over 
500ms. Random reaching movements (B) are 2s long sequences of 
reaching movements between random targets. Colored traces: 
representative samples. Black solid and dashed lines: ensemble mean 
and variance computed across an ensemble of 500 samples. 
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Within the DSPP model, the modulated rate is considered 
as the signal conveyed by the unit. Only rate-conditioned spike 
count variations are considered noise (Churchland et al., 2011). 
Thus, we define the rate-modulations signal-to-noise ratio (RM- 
SNR) as the ratio between the variance of the rate and the 
variance of the Poisson noise (Section Rate-modulations Signal- 
to-noise ratio of DSPP). Theoretical analysis, presented in Section 
Bandwidth Effect and Appendix A, indicates that at short bin- 
widths RM-SNR should increase linearly with the bin-width, 
while at long bin-widths it should saturate. The analysis is 
extended in Section Dead-time Effect and Appendix B to include 
the effect of refractory period (dead-time modified DSPP). 
Section Simulations describes the simulations that are used to 
demonstrate the proposed methods. Section Timescales Analysis 
proposes to capture the trade-off between RM-SNR and update- 
rate by the ratio between RM-SNR and bin-width. The variations 
of this ratio with the bin-width are related to the interaction 
between the refractory properties of the unit and the spectral 
properties of the encoded signals, and the bin-widths at which 
this ratio peaks are related to the timescales of neural encoding. 
The effects of deviations from the DSPP assumption are evaluated 
by analyzing simulated realizations of doubly stochastic Gamma 
processes (DSGPs). 

After demonstrating the analysis on simulated realizations of 
both DSPPs and doubly stochastic Gamma processes (DSGPs), 
it is applied to investigate the timescales of the neural activity 
recorded from cortical motor units during two experiments with 
Monkeys (described briefly in Section Experimental Methods). 
Sections SNR During BMI Experiments and Timescale During 
BMI Experiments present the analysis of spike-trains recorded 
during experiments with BMIs and demonstrate that the average 
timescale agrees well with the experimentally selected bin-width, 
though the timescales of individual units vary. The effect of 
the refractory period is assessed in Section Refractory Effects. 
Most importantly, Section BMI and Training Effects demon- 
strates that the timescales did not change significantly when 
switching to brain control or with training. During locomotion, 
the analysis identified the units whose timescales varied with 
the speed of walking—in agreement with the spectral analysis 
(Section Timescales During Walking). We conclude in Section 
Conclusions and Discussion, by discussing the proposed signal- 
independent timescale analysis (SITA) and its significance for 
investigating the timescales of neural-rate coding under different 
task conditions, and how they are related to the spectral content 
of the encoded signals and the refractory properties of the unit. 


MATERIALS AND METHODS 

MODELING AND ANALYSIS METHODS 

Rate-modulations Signal-to-noise ratio of DSPP 

The Poisson process is the simplest point process in which the 
probability of an event—a spike in the case of neural activ- 
ity, is assumed to be independent of the history of the spike 
train. Inhomogeneous Poisson processes are the simplest point 
processes that can describe rate modulations, i.e., the probabil- 
ity of a spike may change with time (Snyder, 1975; Cox and 
Isham, 1980; Johnson, 1996; Gabbiani and Koch, 1998). Here we 
consider DSPP, which is a subclass of inhomogeneous Poisson 


processes whose rate is modulated by stochastic signals (Snyder, 
1975; Cox and Isham, 1980; Johnson, 1996; Gabbiani and Koch, 
1998). Thus, the statistics of spike trains generated by DSPPs are 
determined by two factors: stochastic changes in instantaneous 
spike-rate and Poisson probability of spike occurrence. 

Considering long intervals of movements, the stochastic sig- 
nals that modulate the instantaneous rate, and thus the instan- 
taneous rate itself, are assumed to be stationary. This is justified 
since we consider long sequences of movements (reaching or 
stepping) that are not synchronized to any event, and thus may 
start at an arbitrary phase of the movement. This approach is 
illustrated in Figure 1 and contrasted with synchronized aver- 
aging, using a simple example where the rate is assumed to 
encode both the position (along 1-dimension) during a reach- 
ing movement and an independent hidden signal. Figure 1A 
illustrates the averaging method, where the same reaching move- 
ment (upper panel) is repeated, and the resulting spike-rates 
(lower panel, depicting a sample of 5 realizations) are synchro- 
nized to movement initiation. The resulting synchronized rate is 
a non-stationary process with time-varying mean (solid black line, 
averaged over a simulated sample of 500 realizations). Figure 1B 
illustrates the proposed approach, showing a sample of five move- 
ment sequences (upper panel) starting at random phases (for 
illustration, each sequence includes four consecutive reaching 
movements between random targets). The resulting spike-rates 
(lower panel) are not synchronized and thus generate a station- 
ary process with constant mean and variance (solid and dashed 
black lines, respectively, based on a sample of 500 realizations). 
The rate-variance (Figure 1B, lower panel) captures the contri- 
butions of both the movement (whose variance is depicted as 
black dashed line in Figure 1B, upper panel), and the hidden sig- 
nal (whose variance is depicted as black dashed line in Figure 1A, 
lower panel). 

Thus, the instantaneous spike-rate A(t) = A9+ X(t) is 
assumed to be a stationary stochastic process with mean rate 
ho and zero-mean rate modulations A(t). In realizations of 
DSPP, the distribution of spike-counts, Nr, in bins of size T is 
determined by the integrated spike-rate during the bin: 


t+T/2 
Ar(t) = 7 (a )do, (1) 
t-T/2 


and its statistics are given by (Snyder, 1975; Zacksenhouse et al., 
2007): 


E[Nr] = E[ArT] 
Var[Nr] = Var[Ar] + E[A7] (2) 


The last equation can be interpreted as a decomposition of the 
total variance in the spike-counts into the variance of the mod- 
ulated rate, Var[Ar], and the variance of the noise E[Ar] = 
E[Nr] (Zacksenhouse et al., 2007). A similar decomposition was 
derived for general point processes from the total variance equa- 
tion (Churchland et al., 2011), however the restriction to DSPPs 
assures that the meaning of the two sources of the neural vari- 
ance is clearly defined (as further detailed in Section Conclusions 
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and Discussion), and facilitates explicit analysis of the effect of the 
refractory period, as detailed in Section Dead-time Effect. 

Considering the modulated rate as the signal, the RM-SNR is 
defined as: 


Var[A 7] 


SNR; = ——., 
7 “ELATI 


(3) 
Using Equation (2), RM-SNR can be estimated from the mean 


and variance of the binned spike-counts as: 


Var[Nr] — ELNr] 
E|Nr] 7 


SNRny = F-1 (4) 


where the hat denotes estimation, and F is the Fano factor, defined 
as the ratio between the variance and the mean of the spike-counts 
(Dayan and Abbott, 2001). Thus, for the simple DSPP case, the 
RM-SNR is the deviation of the Fano-factor from its value for the 
homogeneous Poisson process, where F = 1. Positive RM-SNR 
reflects rate modulations and characterizes irregular spike trains 
(i.e., spike trains whose variance is larger than their mean and 
thus larger than the variance of a homogeneous Poisson process 
with the same mean). Expressing the SNR is advantageous for fur- 
ther analysis of the timescales, as detailed in Section Timescales 
Analysis. 


Bandwidth effect 

The relationship between RM-SNR and the spectrum S; (@) of the 
instantaneous rate A(t) is analyzed in Appendix A. Equation (A1) 
shows that: 


1 oil oe 
SNRr = Tae [. |Gw,(@)|? S;(w)deo (5) 


where |Gw,, (o)| =T (25) is the Fourier transform of the 
rectangular window of duration T (Bendat and Peirsol, 2000). 
Furthermore, under the assumption that the spectrum of the 
instantaneous spike-rate S;(w) is band limited in the range 
[—@max» max], Appendix A derives two asymptotes for RM-SNR 
that hold at short and long bin-widths, respectively. For short 
bin-widths T, i.e., when @max < 27 /T, Equation (A2) indicates 
that RM-SNR increases linearly with the bin-width. For long bin- 
widths T, i.e., when @max >> 27/T, Equation (A3) indicates that 
RM-SNR saturates. 


Dead-time effect 
During absolute refractory period, or dead-time ty, the instanta- 
neous firing rate is zero regardless of the modulating signals. The 
effect of the dead-time on the estimated RM-SNR is detailed in 
Appendix B, and, assuming that the dead-time is short compared 
to the dynamics of the instantaneous rate, an exact expression for 
SN Rn; is derived (Equation B4). 

To gain more insight into the effect of the dead-time, the 
estimated SNR ‘N, is approximated (Appendix B, Equation B9) by: 


P —  SNRr Tq (2 + Tgdo) 
SNRyy (ta) = ° a (6) 
[1 + taAo] [1 + tado] 


The approximation indicates that SNRw;. estimated from a real- 
ization of a dead-time modified DSPP is a scaled and downward 
shifted version of SNRr associated with the original, dead-time 
free, rate. The scaling reflects the effect of the dead-time on the 
integrated rate (Equation B6), and the shift reflects the additional 
effect on the variance of the spike-counts (Equation B3). 


Simulations 

The analysis is demonstrated on simulated realizations of point 
processes with known characteristics. Specifically, the instanta- 
neous rate was generated as a stationary stochastic process by: 
(a) passing a unit variance white Gaussian noise through a 2nd 
order Butterworth filter with known cut-off frequency feu, (b) 
scaling the resulting signal to achieve the desired variance of 
the instantaneous rate, and (c) adding a constant mean rate Apo. 
Finally, simulated spike trains were generated from the resulting 
instantaneous rate using the inverse distribution function tech- 
nique (Johnson, 1996). Spike trains were generated as realizations 
of DSPPs, doubly stochastic Gamma processes (DSGP; Fujiwara 
et al., 2009) with constant shape parameter k, or dead-time 
modified DSPPs or DSGPs with fixed dead-time. In summary, 
the simulated spike trains were characterized by five parameters: 
bandwidth defined by the cut-off frequency feu of the low-pass 
filter, mean (Ap) and variance (V>,) of the instantaneous rate, 
dead-time tg, and the shape parameter for DSGP (x, « = 1 for 
DSPP). 

We note that for homogeneous Gamma processes with k > 1, 
the probability of firing after a spike is reduced, resulting in rel- 
ative refractory period, and the variance of the spike-counts is 
lower than for the homogeneous Poisson process with the same 
mean. In contrast, for homogeneous Gamma processes with k < 
1, the probability of firing after a spike is enhanced, and the 
variance of the spike counts is larger than for the homogeneous 
Poisson process with the same mean. 


Timescales analysis 

While SNR7, and hence SNRN; increase with the bin-width, the 
update rate decreases (Wu et al., 2006) and the reaction time 
increases (Cohen and Newsome, 2009). We quantify the trade-off 
between SNR 'N, and update-rate by evaluating their product, ie., 
the ratio SNRN; /T, as a function of the bin-width T, and charac- 
terize the timescale of neural encoding as the bin-width at which 
this ratio saturates or peaks. This is demonstrated in Figure 2, 
which depicts the variations of SNRn, /T with the bin-width 
for simulated realizations of DSPPs and DSGPs generated from 
the same realization of the instantaneous rate. The instantaneous 
rate was generated with mean Ag = 15 s_!, variance Vi, = 48s~2, 
and cut-off frequency fey; = 1.0 Hz. The realizations analyzed in 
Figure 2A are dead-time free, while the realizations analyzed in 
Figure 2B include dead-time ty = 1 ms. 


Timescale analysis of DSPPs. The estimated SNR ‘N/T from both 
the dead-time free DSPP (Figure 2A, blue line) and dead-time 
modified DSPP (Figure 2B, blue line) closely match the val- 
ues computed from the rate (Equations 3 and B4, respectively, 
black lines). The curves depict the expected effects of the spec- 
tral content and dead-time. At long bin-widths, SNRr should 
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FIGURE 2 | Timescale analysis of simulated realizations of doubly 
stochastic Poisson and Gamma processes (DSPP DSGP) (A) and 
dead-time modified (ty = 1) DSPP and DSGP (B). DSGPs are 
simulated with « =0.85, « =1 (i.e., DSPP), and « = 1.15. All 
simulations were generated from the same instantaneous rate function 
generated with mean Ao = 15s—!, variance V, =48s~2, and cutoff 
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frequency four = 1.0Hz. Estimated SNRw, /T (Equation 4) are compared 
with the values computed from the rate [using Equation (3) for the 
dead-time free DSPP in (A), and Equation (B4) for the dead-time 
modified DSPP in (B)]. Analysis is conducted at bin-widths of 30, 40, 
50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 175, 200, 250, 300, 
500, 750, and 1000ms. 


reach a constant asymptote (Equation A3), so SNR7/T should 
indeed decrease with the bin-width. The additive bias in estimat- 
ing SNR ', from dead-time modified DSPP is constant (Equation 
6), independent of the bin-width, so, even when the spike-trains 
include a refractory period, the effect on the estimated SNR, /T 
would diminish at long bin-widths, as is evident in Figure 2B. 
At the other extreme, i.e., when the bin-widths are short enough 
for the linear asymptote (Equation A2) to hold, SNR7;/T should 
be approximately constant, in agreement with the estimated 
SNRn;. /T for DSPP (blue curve, Figure 2A). Hence, the short 
bin-widths at which SNR; /T saturates are indicative of the 
timescales above which the slower than linear increase in SNR 
does not justify the reduction in update rate. 

Dead-time introduces a constant bias in estimating SNRN; 
(Equation 6), which becomes increasingly significant as SNRr 
diminishes at short bin-widths. Since the bias is negative, the 
estimated SNRn;. /T becomes smaller as the bin-width becomes 
shorter, in agreement with the short bin-width trend of the blue 
curve in Figure 2B. The effect of the dead-time at short bin- 
widths limits the benefit of reducing the bin-width, and results in 
a peak in the SNR, /T curve. The bin-width at which this peak 
occurs maximizes the trade-off between SNRn;. and update rate. 
We also note that below this bin-width the distortion due to the 
dead-time increases significantly. 


Timescale analysis of DSGPs. Applied to doubly stochas- 
tic Gamma processes (DSGPs) with scale parameter (xk) 
larger/smaller than 1, the estimated SNRy,/T under/over 


estimates the value computed from the rate, as demonstrated 
in Figure 2. In realizations of Gamma processes with « > 1, the 
occurrence of a spike has an inhibitory effect, resulting in rela- 
tive refractory period. The estimated SNRn;. /T curve in such a 
case (red line, Figure 2A) or when an absolute refractory period is 
also included (red line, Figure 2B) depict a peak similar to that for 
dead-time modified DSPP. As noted above (Section Simulations) 
realizations of homogeneous Gamma processes with « > 1 are 
characterized by spike-count variance that is smaller than the 
mean. The positive estimated SNRN;. (which implies variance 
larger than mean, see Equation 4) reflects the contribution of the 
variance of the stochastic rate. 

In realizations of DSGPs with « < 1, the occurrence of a spike 
enhances, rather than inhibits, the probability of firing. The esti- 
mated SNR; /T curve in such a case (green line, Figure 2A) 
increases sharply as the bin-width become shorter without sat- 
urating. For the specific parameters selected here, the excitatory 
recovery after a spike dominates the inhibitory effect due to the 
dead-time, and a similar increase is observed even with dead-time 
(Figure 2B). 


Timescales from SNRn,/T curves. In summary, the analysis and 
simulations indicate that: (i) DSPPs are characterized by decreas- 
ing SNRw,,/ T curves that saturate at short bin-widths. In those 
cases, the bin-widths at which the SNRw,,/ T curve saturates are 
indicative of the timescales above which the slower than linear 
increase in SNR does not justify the reduction in update rate. 
(ii) dead-time modified DSPP or DSGP with relative refractory 
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period (k > 1) are characterized by SNRn;. /T curves that peak 
at some bin-width. The bin-width at which the SNR, /T peaks 
maximizes the trade-off between SNRN; and update rate, while 
limiting the distortion due to the refractory effect. (iii) DSPGs 
with excitatory recovery (« < 1) are characterized by decreasing 
SNRn;. /T curves that remain convex even at short bin-widths 
without saturating. In those cases, the appropriate time-scale 
is not evident from the proposed analysis, since the higher 
SN Ry,/T at short bin-widths reflect recovery effects rather than 
actual rate modulations. Further analysis is needed to select the 
bin-width that optimizes the trade-off between high SNRN; /T 
and small distortion due to the excitatory recovery effect. 


Variance, bandwidth and dead-time effects. The effects of the 
variance, bandwidth and dead-time are further evaluated in 
Figure 3, based on the approximation derived in Equation 6. 
Figure 3A establishes that the approximated SNR; /T (dashed 
black line) captures well the main effect of the dead-time. Hence, 
the above effects are evaluated by approximating SNRN,. directly 
from the simulated stochastic rate functions. Unless otherwise 
specified, the nominal parameters of the stochastic rate functions 
are: mean rate 49 = 15s~!, variance V; = 48s~2, bandwidth 
feut = 1 Hz, and dead-time tg = 0.7 ms. 

Figure 3B demonstrates that the major effect of increasing 
the variance (from V;, = 24s~? to V, = 48s~2 and V;, = 72s?) 
is to increase the SNR. This is also accompanied by a shift in 
the peak location toward shorter bin-widths, due to the inter- 
action between SNR and dead-time (Equation 6). Figure 3C 


demonstrates that as the cutoff frequency fc, increases from 
0.5 Hz to 1.0Hz, and 1.5 Hz, SNRn,./T peaks at shorter bin- 
widths, capturing well the reduction in the relevant timescales. 
Figure 3D demonstrates that as the dead-time increases to 0.7 
and 1.4ms, SNRN; /T peaks at longer bin-widths. This trend 
reflects the increasing effect of the dead-time on the magnitude 
of the negative bias in Equation (6) (see note following Equation 
B9). Thus, for dead-time modified DSPP, the bin-width at which 
SNRy;/ T peaks reflects the conflicting effects of the bandwidth 
of the encoded signals and the refractory period. 


EXPERIMENTAL METHODS 

The analysis tools developed above to estimate the RM-SNR of 
spike-trains and evaluate their timescales are applied to spike- 
trains recorded form cortical units in two experiments with 
Monkeys: (a) BMI experiments and (b) Locomotion experiments. 


Ethics statement 

All experiments were conducted with approved protocols 
from the Duke University Institutional Animal Care and Use 
Committee and were in accordance with the NRC/NIH guidelines 
for the Care and Use of Laboratory Animals. 


BMI experiments 

As detailed in Carmena et al. (2003), the BMI experiments were 
conducted with macaque monkeys and included three control 
modes: (i) pole control, (ii) brain control with hand move- 
ments (BCWH), and (iii) brain control without hand move- 
ments (BCWOH). During pole control the monkey controlled the 
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FIGURE 3 | Approximated SNRy,/T for dead-time modified DSPP 
(Equation 6) in comparison with the SNRy, /T computed from the 
rate (Equation B4) or estimated from the spike-counts (A). The 
approximation is used to demonstrate the effects of the variance 
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(B) bandwidth (C) and dead-time (D). Unless otherwise specified, the 
nominal parameters of the stochastic rate functions are: mean rate 

Apo = 15s~', variance V, = 48 s~2, bandwidth fou; = 1Hz, and dead-time 
Tq = 0.7 ms. 
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cursor on the screen using a hand held pole. Data from pole con- 
trol was used to train a linear filter to predict the velocity from the 
neural activity. During brain control the cursor was controlled 
to follow the velocity predicted from the recorded neural activ- 
ity using the previously trained linear filter. Initially, the monkey 
continued to move its hand during brain control (BCWH), but 
starting at the 7th session, the monkey stopped moving the hand 
(BCWOH). Spike trains were recorded from single and multi- 
units (see Section Refractory Effects for more details) in the 
primary motor area (M1), the pre-motor area (PMd), the supple- 
mentary motor area (SMA), and somato-sensory area (S1). Here 
we focus on significantly modulated units (n = 163), ie., those 
units for which the null hypothesis that their ISI distribution was 
generated from a homogeneous Poisson process can be rejected at 
a = 0.05 significance level (Zacksenhouse et al., 2007). 


Locomotion experiments 

During the locomotion experiments, rhesus macaques either 
stood or walked bipedally on a treadmill in three different speeds, 
12.5, 25, and 50 cm/s, as detailed in Fitzsimmons et al. (2009). 
Spike trains were recorded from single and multi-units (66 and 
34%, respectively) in regions associated with the representation 
of the lower limbs in the primary motor (M1) and somatosensory 
(S1) cortices. 


RESULTS 

SNR DURING BMI EXPERIMENTS 

Figures 4A,C depict the mean SNRn,, and mean normalized 
SNRn,, estimated from spike-trains of significantly modulated 
cortical units (n = 163) recorded during a BMI experiment, as 
a function of the bin-width T. Assuming ergodicity, Equation 
(4) was used to estimate the SNRn,. of each unit from its 
spike-counts statistics in non-overlapping windows of 2 min. 
The mean SNRN; in Figure 4A was derived by first computing 
ensemble mean (over 1 = 163 units) at each window, and then 
computing the mean and standard deviation over 10 consecu- 
tive windows in each of the experimental modes (pole control 
and brain control with and without hand movements). Thus, 
error bars in Figure 4A depict the standard deviations of the 
ensemble-means over the 10 windows of time. Variations across 
the ensemble of units are described in Figure 4C, which depicts 
the ensemble mean and standard deviations of the normalized 
SNRn;- The normalized SNRn,, for each unit was computed 
by first averaging across all windows, and then normalizing by 
a scaling factor. The scaling factor was selected so the nor- 
malized SNRN; at bin-width of 300ms is the same for all 
units and equals the original ensemble mean. Thus, error bars 
in Figure 4C depict the standard deviations of the normalized 
SNR; across the ensemble of significantly modulated units 
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FIGURE 4 | Mean estimated SNRn, (A) and normalized SNRn, (C), 
and the corresponding SNRy,/T (B,D) of neural activity recorded 
from significantly modulated units (n = 163) during BMI experiments. 
BMI experiments included pole control, brain control with hand 
movements (WH), and without hand movements (WOH). SNRwy, was 
computed from the spike-train of each unit in non-overlapping windows of 
2-min. (A,B) are based on first averaging over the units at each window, 
and then computing the mean and standard deviation over 10 consecutive 
windows in each of the experimental mode. Hence, errorbars in 
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(A,B) mark the standard deviations across 10 consecutive windows. (C,D) 
are based on first averaging the SNR, of each unit across all windows, 
and then scaling it so the normalized SNRyy, at bin-width of 300 ms is the 
same for all units and equal the original ensemble mean. Ensemble mean 
and standard deviation of the normalized SNRw, were computed across 
units. Hence, error bars in (C,D) mark the standard deviations across the 
ensemble of units (resulting in zero standard deviation at 300 ms, which 
was used for normalizations). Dashed lines in (A) depict the linear fits for 


short bin-widths, over the range 50-150 ms. 
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(resulting in zero standard deviation at 300 ms, which was used 
for normalizations). 

Figures 4A,C demonstrate that the mean estimated SNR; 
increases approximately linearly at short bin-widths and 
approaches a constant level at long bin-widths, in agreement with 
the expected asymptotes (Equations A2 and A3). Another strik- 
ing feature is that at each bin-width, the mean SNRN;. in brain 
control is always higher than that in pole control; and is high- 
est in brain control without hand movements. This extends our 
previous results (Zacksenhouse et al., 2007) for T = 100 ms—the 
bin-width used for operating the BMI. 


TIMESCALE DURING BMI EXPERIMENTS 

Figures 4B,D demonstrate that the mean SNRn,, /T and mean 
normalized SNRn;, /T increase at short bin-widths, reach a wide 
peak around 100 ms, and decrease at long bin-widths. This pat- 
tern agrees well with the analysis and approximation of the 
SNRw;. /T of dead-time modified DSPP, and with estimations 
derived from the simulated spike-trains of dead-time modi- 
fied DSPPs (or DSGPs with « < 1, with or without dead-time) 
depicted in Figure 2. 

Focusing on individual units from PMd and M1, the SNR N/T 
curves shown in Figure 5 demonstrate two typical patterns: (i) 
curves with a wide peak around 100 ms (Figures 5C,D), simi- 
lar to the average curves in Figures 4B,D and to the curves for 
dead-time modified DSPP or DSGP with x < 1, in Figure 2, or 
(ii) decreasing curves (Figures 5A,B), similar to the curves for 
DSPP or DSGP with « > 1 in Figure 2. The second type could 
be further divided depending on whether the curve saturates at 
short bin-widths as for DSPPs, or remains convex as for DSGPs 
with « > 1. 

Units were classified as exhibiting a wide peak around 100 ms 
if the peak SNRn,./T was in the range of 50-150 ms, and as 


exhibiting decreasing SNRn;. /T if the peak was below 50 ms. 
Note that SNRN; /T of few units peaked at bin-widths longer 
than 150 ms, and were not included in either group. To assure 
proper peak-detection, we restricted this classification and fur- 
ther analysis of the timescales to the significantly modulated units 
whose maximum SN. Ry, /T during pole control was larger than 
0.5s~! (n = 109). Of those units, 59.7, 64.2, and 61.5% exhib- 
ited a wide peak around 100ms during pole control, BCWH 
and BCWOH, respectively, while 25.7, 21.1, and 22.0% exhib- 
ited decreasing curves, respectively. The second group was further 
divided depending on whether the decreasing curve was convex at 
short bin-widths, i.e., whether u2 < (uy + u3) /2 where 11, u2, U3 
are the mean SN Ry, /T in the bin-width intervals 30-50, 60-80, 
and 90-100 ms, respectively. About 15% of the curves were con- 
vex at short bin-widths (17.4, 11.0, and 15.6%, in pole control 
BCWH and BCWOH, respectively). In those cases, the time-scale 
analysis is not conclusive, and is either not appropriate since the 
recovery function is dominated by excitatory effects (as clarified 
in Section Timescales Analysis) or should be extended to shorter 
bin-widths. 

The SNRy,/T estimated form spike-trains recorded during 
BMI experiments (Figures 4B,D, 5) vary with the bin-width in a 
similar way to the SNR ‘N/T of simulated spike-trains (Figure 2). 
Decreasing curves, which saturate at short bin-widths (Figure 5B, 
pole and BCWH) agree well with the assumption that the spike- 
trains are realizations of DSPP while decreasing curves that 
remain convex at short bin-widths (Figure 5A, pole and BCWH) 
agree well with the assumption that the spike-trains are real- 
izations of DSGP with « < 1. Curves that peak (Figures 5C,D) 
agree well with the assumption that the spike trains are realiza- 
tions of dead-time modified DSPP or DSGP with « > 1 (with 
or without dead-time), which are characterized by relative, rather 
than absolute, refractory period. 
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FIGURE 5 | Estimated SNRy,/T as a function of the bin-width for four representative cortical units in PMd (A,C) and M1 (B,D), during pole control, 
and brain control with and without hand movements, WH and WOH, respectively. 
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Refractory effects 

To assess the potential contribution of refractory effects, we 
estimated the refractory interval from the ISI distribution. The 
minimum ISI in all units was 0.675 ms, and most units (95% 
of the significantly modulated units) had minimum ISI of less 
than 1 ms. Since a single ISI is prone to measurement errors, we 
estimated the dead-time as the interval which was exceeded by 
99% of the ISIs in at least one mode of operation. Of the 109 
units described above, 50% had dead-time greater than 1.6 ms, 
and thus could be considered single-units (Hatsopoulos et al., 
2004). The percent of units having decreasing curves was sim- 
ilar, independent of whether the dead-time was short or long 
(below or above 1.6ms); but the percent of units having time- 
scales longer than 150 ms was larger for units with long dead-time 
(22.4% on average, across all modes, compared with 8% for 
units with short refractory interval). Furthermore, the bin-widths 
at which SNRN, /T peaked had a moderate positive correlation 
with the product of the dead-time and the mean rate (coeffi- 
cient of correlation of 0.4, 0.42, 0.48 in pole control BCWH and 
BCWOH, respectively). Both the positive correlation and its mod- 
erate value are in agreement with Equation (6), which indicates 
that this product reduces the SN Rn, (and hence shifts the peak of 
the SNRw,,/ T toward longer bin-widths, as clarified in Section 
Timescales Analysis), but that SNRn,. is also affected by other 
variables (i.e., SNRr). 


BMI AND TRAINING EFFECTS 

Figure 5 indicates that the bin-width at which the SNRN, /T 
peaks remains approximately the same even after switching to 
brain control. Specifically, the SNRw;. /T curves in pole control, 
BCWH and BCWOH peak at: 30, 40, and 30 ms for the PMd unit 
in panel (A), 130, 150, and 140 ms for the PMd unit in panel (B); 
30, 40, and 30 ms, for the M1 unit in panel (C); and 150, 150, 
and 110 ms for the M1 unit in panel (D). The distribution of 


the bin-widths at which the estimated SN. Ry, /T peak and how 
they change when switching to brain control is summarized in 
Figure 6. The bin-width at which the estimated SNRN; /T peaks 
during pole control is similar to (within 50 ms of) the bin-width 
at which it peaks during brain control for most of the units (80.7 
and 67.9% for BCWH and BCWOH,, respectively). 

The spectral content of the spike-counts (in 100 ms bins) of 
the four units whose timescales were analyzed in Figure 5, are 
depicted in Figure 7. Specifically, the power spectrum density 
(PSD) was computed from the spike-counts in bins of 100 ms 
after subtracting the mean and expressed in dB. The main effect 
of the transition to brain control is the increase in power, which is 
higher in brain control than in pole control and is highest in brain 
control without hand movements. This is consistent with the 
higher RM-SNR in brain control compared to pole control. The 
overall bandwidth of the activity of each unit remains relatively 
similar across the different BMI control modes [even when, as in 
panel (D), the PSD of the activity during pole control portrays a 
small peak which is surpassed by the higher overall power during 
brain control]—consistent with the relatively similar shapes and 
peak locations of the different SNR; /T curves for each unit in 
Figure 5. 

Figure 8 demonstrates that the shape of the SNRN; /T curves 
is also invariant to training. At later sessions, as training pro- 
gresses, the SNR at each control mode and bin-width decreases (as 
detailed in Zacksenhouse et al., 2007) for bin-width of 100 ms). 
However, the effect of bin-width is similar and the SNRw,,/ T 
curves peak at a similar bin-width in all sessions. 


TIMESCALES DURING WALKING 

Figures 9, 10 depict the estimated SNR N/T and the spectrum of 
representative cortical units in the motor leg area while the mon- 
key was standing (black) or walking at increasing speeds (12.5, 
25, and 50 cm/s). The SNRn,. /T curves in Figures 9A,B peak at 
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FIGURE 6 | Distribution of timescales during BMI experiments. 
Comparison of bin-widths at which SNRw,/T curves peak during pole control 
and during either brain control with hand movements, (VVH, A) or brain 
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control without hand movements (WOH, B). Only significantly modulated 
units whose peak SNRy,/T during pole control was larger than 0.5 s—' were 
included (n = 109). 
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FIGURE 7 | Power spectrum density (PSD) of the neural activity analyzed in Figure 4. The spectrums were computed from spike-counts in 100 ms bins, 
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FIGURE 8 | Mean estimated SNNRy,/T across the population of 
significantly modulated cortical units recorded in three different BMI 
sessions during pole control (PC), and brain control with and without 
hand movements (BCWH and BCWOH, repectively) as a function of 
the bin-width. The three sessions include: an early session (3rd day); a 
mid session (7th day), and a late session (10th day). The 7th day is the first 
one in which BCWOH was performed, and in subsequent sessions BCWH 
was not perfomrned. 


progressively shorter bin-widths as the speed of walking increases. 
During standing, the peak SNRN;. /T is shallower and appears at 
longer bin-widths than during walking. Figures 10A,B indicate 
that the spectrums of the spike-counts (in 100 ms bins) of these 
spike-trains are dominated by single spectral lines that appear 


at higher frequencies as the speed of walking increases. During 
standing, the dominating spectral line appears at very low fre- 
quencies and is smaller compared to the spectral lines during 
walking. Thus, this simple case, in which the spectrums are dom- 
inated by single spectral lines, helps to demonstrate the expected 
effect of changes in the spectral content of the modulating sig- 
nals on the timescale. However, the timescale emerges from the 
interaction between the spectral content and the refractory prop- 
erties of the individual unit. Hence, even though the dominating 
spectral lines appear in the same frequencies (Figures 10A,B) the 
timescales are unit specific (Figures 9A,B). 

In contrast, the neural activity from other units in the leg area 
are characterized by SNRN; /T curves that either decrease with- 
out peaking (Figure 9D) or depict a peak that does not shift as 
expected with the speed of walking (Figure 9C). In both cases 
the neural activity during standing is characterized by higher 
SNRN; /T than during walking. The corresponding spectrums 
(Figures 10C,D) indicate that the power of the spike-counts 
recorded from these two units is indeed larger during standing 
than during walking. The spectral lines associated with the speed 
of walking (at the frequencies in which they appear in the spec- 
trums depicted in panels A and B) are very small. Finally, for 
both units the neural activity recorded during running has more 
power in the low-frequency range than during walking. This may 
explain the unexpected shift in the peak of the SNR/T toward 
longer rather than shorter bin-widths when the Monkey is run- 
ning compared to walking (Figure 9C). The convex shape of the 
SNRy,,/ T curve in Figure 9D even at short bin-widths suggests 
that the activity of this unit is dominated by excitatory recovery 
effects, and the timescale is not conclusive. Nevertheless, since the 
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FIGURE 9 | Estimated SNRy,/T as a function of the bin-width for four representative cortical units (A—-D) in the motor leg area while the monkey is 
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FIGURE 10 | Power Spectrum Density (PSD) of the neural activity analyzed in Figure 9. PSDs were computed from the binned neural activity with 100 ms 
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SNRn,, /T curves are similar during walking and running (inde- 
pendent of the speed) but much higher during standing, it can 
be concluded that signals associated with the frequency of walk- 
ing have little effect on the spike-rate of these two units, and that 
their activity is more strongly modulated during standing. 


Additional analysis (not shown) indicates that the mean 
SNRn;. /T and mean normalized SNRN, /T across significantly 
modulated units with peak above 0.5s~! (n = 48) depict a wide 
peak around 50-110 ms, in most cases. The only exception is the 
normalized SNR, /T during standing, which is highest at the 
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shortest bin-width analyzed (25 ms). Around the peak, the mag- 
nitude of the SNRN; /T are similar independent of the speed so 
the curves overlap each other. At longer bin-widths SNRw,,/ T 
during slow walking is higher than during fast walking or run- 
ning, and is highest during standing. 


CONCLUSIONS AND DISCUSSION 

SNR ESTIMATION 

We suggest a computational framework for estimating the SNR in 
spike trains without relying on any assumption about the encoded 
signals. The suggested approach captures all the encoded sig- 
nals, including hidden and internal signals, and not only those 
controlled experimentally. The proposed approach has two lim- 
itations: (i) it accounts also for possible task irrelevant rate 
modulations, and (ii) the estimation method is derived under the 
assumption that the spike-trains are realization of DSPP. The first 
limitation is a necessary side-effect of capturing the hidden or 
internal signals. The second limitation is partially relaxed by con- 
sidering dead-time modified DSPP and doubly stochastic Gamma 
processes. 

The dependency of RM-SNR on bin-width was first analyzed 
for DSPP and two asymptotes were derived for bin-widths that are 
either very short or very long compared with the inverse of the 
bandwidth. In particular, it was shown that RM-SNR increases 
linearity at short bin-widths and reaches saturation at long bin- 
widths. Extending the analysis to dead-time modified DSPP, it was 
shown that the estimated RM-SNR is reduced by both scaling and 
shifting factors that do not depend on the bin-width. The bias 
effect becomes more significant as the RM-SNR becomes smaller 
at shorter bin-widths. Using simulations of doubly-stochastic 
Gamma processes, we also evaluate other recovery effects in which 
the occurrence of a spike temporarily and partially inhibits (as in 
relative refractory period) or enhances the probability of firing. 


INHOMOGENEOUS vs. DOUBLY STOCHASTIC POINT PROCESSES 
Doubly stochastic point processes are inhomogeneous point pro- 
cesses with stochastic rate function. The alternative model for 
describing rate modulations is the inhomogeneous point pro- 
cess with a deterministic rate function (Cox and Isham, 1980; 
Kass and Ventura, 2001; Nawrot et al., 2008). The determin- 
istic model is based on the assumption that the rate function 
depends solely on the experimentally controlled conditions, and 
thus that it can be estimated from the post-stimulus time his- 
togram. This assumption might be valid for peripheral neurons 
that are affected almost exclusively by the experimentally con- 
trolled stimulus, but less for cortical neurons which may encode 
additional hidden signals, not directly under experimental con- 
trol. These signals may vary from trial-to-trial and thus can only 
be modeled as stochastic. During movements, motor planning 
(Churchland et al., 2006a,b) and prediction errors (Wolpert and 
Ghahramani, 2000; Shadmehr and Krakauer, 2008), for example, 
may vary from trial-to-trial, while in decision making the accu- 
mulated evidence (while potentially sharing a common trend), 
may vary from trial-to-trial (Churchland et al., 2011). In these 
cases, doubly stochastic point processes are essential for captur- 
ing the variance of all the signals that might be encoded in the 
neural activity. 


VARIANCE DECOMPOSITION 

Under the assumption that spike-trains are realizations of DSPPs, 
the variance of the spike-counts is decomposed into two terms 
associated with the signal and noise. A similar decomposition was 
developed in Churchland et al. (2011) where the resulting two 
terms were described as the variance of the underlying “inten- 
sity command” and the variance of the neural activity given 
that command. That decomposition was applied to uncover how 
trial-to-trial neural variability changes during decision making 
and reveal the underlying nature of neural processing. While the 
application and some of the analysis details differ (as discussed 
below), we share the most important hypotheses: (i) the neu- 
ral activity may represent hidden signals, ie., signals that are 
not directly controlled and hence may vary from trial to trial, 
(ii) decomposition of the variance of neural activity facilitates 
estimating the variance of the underlying rate, including modula- 
tions by those hidden signals, and (iii) variations in the variance 
of the underlying rate with time (or task) may reveal important 
information about the neural processing. 

The decomposition developed in Churchland et al. (2011) is 
based on the total variance equation, which holds for any dou- 
bly stochastic point process. The total variance is decomposed 
into the variance of the conditional expectation (the variance of 
the expected spike-count given the rate parameter and the point 
process variance (the expectation of the conditional variance). 
Under the restriction to DSPP made here, the first term equals 
the variance of the rate parameter, which is assumed to encode the 
modulating signals. Otherwise, the relationship of the first term 
to the variance of the rate parameter is more complex. In par- 
ticular, for dead-time modified DSPPs and for DSGPs, the first 
term is proportional to the variance of the rate-parameter but 
not equal to it (see Equations B1 and Bé6 for the dead-time mod- 
ified DSPP). Hence, the restriction to DSPP made here assures 
that the meaning of the first term is clearly defined. Furthermore, 
it enables explicit analysis of the effect of the refractory period 
on both terms. Another major difference is that the analysis in 
Churchland et al. (2011) was applied to estimate the synchronized 
trial-to-trial variance in the underlying rate-process (intensity 
command) at each time along the trial. In contrast, we applied the 
analysis to estimate asynchronous variance in the underlying rate 
process. 


TIMESCALES OF NEURAL ACTIVITY 

As bin-width increases, SNR increases but update-rate is com- 
promised (Wu et al., 2006). This trade-off is captured by the 
ratio of SNR to bin-width, SNRr/T. Thus, the bin-width at 
which SNR7/T peaks optimizes the trade-off between SNR and 
update rate and is suggested as a criterion for characterizing the 
timescales of neural rate-coding. 

Our analysis suggests that the estimated SN Ry, /T is affected 
by both the spectral content of the encoded signals and the 
recovery properties of the unit. The effects of the bandwidth 
and absolute refractory period were analyzed theoretically and 
demonstrated via simulations of DSPPs and dead-time modi- 
fied DSPPs. Their conflicting effects give rise to a peak in the 
SNRn; /T curve, which characterizes the timescale of neural 
rate-coding. 
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DEVIATIONS FROM THE DSPP ASSUMPTION 

The DSPP assumption is violated when the firing rate depends on 
the history of the spike train. In these cases, the estimated SNRN; 
may differ from the actual SNRr, due to history-dependent mod- 
ulations of the instantaneous rate. We restrict the analysis to the 
simple, yet common case, of renewal point processes, in which 
the dependency on the history is captured by the time elapsed 
since the last spike. The effect of absolute refractory period was 
analyzed theoretically, and was shown to result in a peak in 
the SNRw,,/ T curve. Thus, the dead-time limits the benefit of 
reducing the bin-width, since the estimated SNRN; becomes 
increasingly distorted. The bin-width at which the peak occurs 
maximizes the trade-off between SNRN; and update rate, and 
characterizes the timescale of neural rate-coding. 

Using simulations we also demonstrated that relative refrac- 
tory periods have similar effect, resulting in SNR, /T curves 
that are also characterized by a peak. So the timescales are sim- 
ilarly characterized by the bin-widths at which the SNRN; /T 
curves peak. In contrast, when the recovery period is charac- 
terized by enhanced, rather than inhibited, probability of firing, 
the SNRN; /T over-estimates (rather than under-estimates) the 
SNRr/T. Thus, as the bin-width decreases, SNRw,,/ T increases 
without peaking or saturating. In those cases, the appropriate 
time-scale is not conclusive from the proposed analysis, since 
the higher SNRw,,/ T at short bin-widths reflect recovery effects 
rather than actual signal modulations. Further analysis is needed 
to select the bin-width that optimizes the trade-off between high 
SNRn,, /T and small distortion due to the excitatory recovery 
effect. 


TIMESCALES DURING BMI EXPERIMENTS 
Estimating RM-SNR from neural activity recorded during BMI 
experiments, we observe that while its magnitude depends on the 
BMI mode and on training, the timescale revealed by the peak of 
the SNRN; /T curve remains the same. 

The observation that RM-SNR increases when switching to 
brain control indicates that the variance of the rate increases— 
and thus that the variance of some of the encoded signals 
increases. The observation that the timescales remain invariant 
suggests that those signals are task-relevant, and we are currently 
investigating the hypothesis that they include state estimation 
and control signals, within the framework of optimal control. 
However, their exact nature and decoding remains an open issue. 

Most of the analyzed SNR N/T curves (62% on average across 
the different control modes, n = 109) were characterized by a 
wide peak at 50-150 ms. For this class of curves (and for the 
15% of curves that peak at longer bin-widths), decreasing the bin- 
width below the peak may compromise decoding, unless the effect 
of the refractory period is accounted for. Indeed BMI applica- 
tions based on spike-counts used bin-widths in the range between 
30 and 100 ms (Wessberg et al., 2000; Serruya et al., 2002; Kim 
et al., 2008; Velliste et al., 2008), consistent with the range of 
timescales revealed here. In particular, the BMI experiments ana- 
lyzed here were conducted with 100 ms bins (Carmena et al., 
2003) within the wide peak of the estimated SNR, /T curve 
depicted in Figure 4B. Our analysis suggests that this selection 
provides a good trade-off between the SNR and update rate. 


Curves whose peak was below 50 ms were divided into those 
that were convex or concave at short bin-widths (15 and 8% 
on average, respectively). Concave shape at short bin-widths is 
expected from SNR, /T curves derived from DSPPs, where this 
ratio should saturate at short bin-widths as detailed in Section 
Timescales Analysis (and Figure 2A). In this case, the range of 
bin-widths at which the curve saturates is indicative of the rel- 
evant timescales for neural decoding. At longer bin-widths, the 
slower than linear increase in SNR does not justify the reduction 
in update rate. Convex SNR; /T are expected when the unit has 
excitatory recovery period (in which the probability of firing is 
enhanced). As detailed above, the timescale analysis in this case is 
not conclusive. 

Since the relevant timescales vary across individual units, 
it might be beneficial to apply different bin-widths when bin- 
ning different units. This may explain the potential advantage 
of multi-resolution binning (Kim et al., 2005). Recent decod- 
ing methods are based on point process filters that operate at 
5 ms bins (Shanechi et al., 2013a,b). The BMI tested in Shanechi 
et al. (2013a) was based on multi-unit recording, where the dead- 
time effect might be negligible, and hence the timescales could 
be short. Furthermore, point process filters are based on estimat- 
ing the conditional probability of the spike-counts given the rate, 
which is computationally more efficient when the bin is small 
(and the number of spikes that may occur with significant prob- 
ability is limited). Hence, the choice of small bins may reflect 
computational constraints. The analysis conducted here suggests 
that unless recovery effects are negligible or accounted for, very 
short bin-widths may compromise the trade-off between SNR 
and update-rate. 


TIMESCALES DURING LOCOMOTION EXPERIMENTS 

The overall shape of SN. Rn, /T curves estimated from spike trains 
recorded during walking is similar to those derived from the spike 
trains recorded during the BMI experiments. However, the peak 
location in the locomotion experiments varied with the task, at 
least for some units, and shifted to lower bin-widths when the 
speed of walking increased. This is in agreement with the expected 
change in the bandwidth of encoded signals associated with the 
experimentally controlled speed of walking. Indeed, units whose 
timescales vary in this way are characterized by PSDs dominated 
by a single peak at the frequency of walking. Thus, the locomotion 
test-case demonstrates the expected effect of changes in the spec- 
tral content of the modulating signals on the timescales. However, 
timescales depend not only on the spectral content but also on the 
refractory properties of the units. So even if the spectral content 
is similar the timescales are unit-specific. 

In summary, we developed a method for assessing the SNR and 
timescales of neuronal activity independent of the encoded sig- 
nals and observe that many units depict well-defined timescales 
of the order of tens to hundreds msec. The timescale emerges 
from the interaction of the recovery properties of the unit and 
the spectral content of the modulating signals. In particular, 
the variations of the estimated SNRN; /T at short bin-widths 
is related to the recovery properties of the unit. During a 
BMI experiment, the average timescale across the population 
agrees well with the experimentally selected bin-width, though 
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individual units depict different timescales. Furthermore, while 
SNR increases after switching to brain control and decreases 
with training, timescales remain similar. During locomotion, 
the analysis identified units whose timescales varied consistently 
with the experimentally controlled speed of walking, though 
the specific timescale reflected also the recovery properties of 
the unit. Hence, the proposed method provides a hypothe- 
sis free tool for investigating possible contributions of hidden 
signals, not under experimental control, to neural modula- 
tions and the effect of task conditions on their magnitude and 
timescales. 
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APPENDIX A: BANDWIDTH EFFECT 

The binning process is equivalent to passing the instantaneous 
spike-rate through a finite time integrator followed by down- 
sampling and spike-counting (see Bair et al., 1994 for a similar 
formalization of the recording of spike-trains with finite time res- 
olution). Specifically, the integrated spike-rate A7r(t) = AoT + 
Ar( t), where Ar(t) = = a fe A(o )do, is obtained by convolving 
the time-dependent component of the instantaneous spike-rate 
Mo) with a rectangular window of duration T: Ar(t) = X(t) ® 
Wr(t) [® denotes convolution, and W(t) is a rectangle signal 
of duration T (Bair et al., 1994)]. The effect of down-sampling 
will be considered later. For now we note that the spectrum of 
7 denoted by Sx (w), is related to the spectrum S; (w) of A(t) 
by: Sx = |Gw; (o)|? 5; (@), where |Gw, (w) = — 7 (salar) 
is the a transform of the rectangular window (Bendat and 
Peirsol, 2000). The variance of A7(t) can be computed from 
the spectrum Sx (@) as: Var[Ar(t)] = + . 5x (@)dw, while 
its mean is E[A r(t)] = EL[Nr(t)] = AoT. Using Equation (3), the 
SNR of the binned spike-counts is given by: 


1 1 
SNR; = ——=— (Al) 


hoT 2a f Gw(o)l S;(w)do 
Assuming that the spectrum of the instantaneous spike-rate 
S;(@) is band limited in the range [—@max, @max], the SNR 
reaches two asymptotes for short and long bin-widths. For 
short bin-widths T, ie, when @max < 27/T, the transfer 
function of the rectangular window |Gw,( w)| in the range 
[—@max, @max] can be approximated by |Gw,(0)| = = T, so, from 
Equation (A1): 


1 T? (max a 
SNR; = ——-— S-(w@)do = T, 


A2 
AoT 20 ( ) 


—@max 


1 max S;(@)dw is the power of the time- 


where Px; = 5— he 

dependent component vii the instantaneous rate. Hence, at short 
bin-widths, the SNR of spike-counts generated from DSPP with 
band-limited instantaneous rate should increase linearly with the 
bin-width T. 

For long bin-widths T, i.e., @max >> 27 /T, the Fourier trans- 
form of the rectangular window is concentrated around zero, 
so only low frequencies pass the averaging process and affect 
the average rate. Denoting by So the spectrum of the instanta- 
neous rate at low frequencies, Equation (A1) implies that at long 
bin-widths the SNR saturates at: 


So 


SNRT => 
ad T—>00 ler Qn 


Pew, (w)|’ do = a = —. (A3) 


Binning is usually performed in non-overlapping windows, so 
the binned spike-counts are related to the samples of the inte- 
grated spike-rate taken at sampling interval T. At short bin- 
widths, the Nyquist frequency Ny = @sampling/2 = 1/T satisfies 
the Nyquist criterion wyy > ®max. Hence, there is no aliasing, 
and Equation (A2) holds, implying that the SNR of binned spike 
counts should increase linearly with the bin-width. At long bin- 
widths, the Nyquist criterion is not satisfied, and the side bands 


of the spectrum of the rectangular window |Gw,(@) a fold back 
into the main lobe. Nevertheless, as long as the spectrum is 
constant over most of the side-lobes, the total power in the 
main lobe (including all the folded back side-lobes) is the same 
as the integral computed in Equation (A3) (Bair et al., 1994). 
Hence the SNR of binned spike counts should saturates at long 
bin-widths. 


APPENDIX B: SNR OF DEAD-TIME MODIFIED DSPP 

Assuming the dead-time is short compared to the dynam- 

ics of the instantaneous rate, the statistics of the spike-counts 

depend 7 as dead-time modified integrated rate: Ap = 
sa ee i+ : en ;do, and are given by [see Equations 3 and 9 in 

Vannucci and Teich (1981)]: 


E[Ar] 
t+T/2 


avs = e[(any']+5| 


E[Nr] = (B1) 


Mo) 
(i+ a] 7 


So: 


t+T/2 


Var[Ny] = Var[Ar] +E / ee cae (B3) 
aa 1/2 [1+ taA(o)]? 


Substituting Equations (B1) and (B3) in Equation (4), the esti- 
mated SNR from a dead-time modified DSPP is: 


t+T/2 Mo) mi 
SNRy,,.(Ta) = Valea +e [fe T/2 de | —E[Ar] 
T- 
E[Ar| i 


The SNR associated with the dead-time modified integrated rate, 
SNRyz, can be defined in a similar way to the definition of SNRr 
for the original integrated rate (Equation 3): 


(B5) 


For tgA(o) - 1, the Dee Geeuing ma can be approximated 
t+T/2 


by: Are itun je Ti * (a)do = rts 7? 80 its statistics can be 
approximated by: 
E[ Ar] a ElAr) (B6) 
1+ tyro 
bs Var [A 
Var[A7] & neal (B7) 
[1 + tgAo] 
Hence, the SNRr can be related to SNR¢ by: 
_ SNRr 
SNRr(tg) = ———— (B8) 
me’ TL tah 


Using a similar approximation, the second term in Equations 
(B2—B4) can be approximated as: 
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(ie Ao) ~ E[Ar] 
E do | = 
r-T/2 [1+ tgd(o)]> [1+ tado]? 


Substituting these approximations in Equation (B4) results in an 
approximated relationship between the estimated SNR from a 
dead-time modified DSPP and the SNR of a dead-time free DSPP 
having the same instantaneous rate: 


(B9) 


z = TdAo (2+ Tar SNR 
SNRy, (tg) & SNRy — TH06 a» a a4 
[1+ tado] [1 + taAo] 
TqAo (2 + Tr 
_ Taho ( di 0) (B10) 
[1 + Tap | 


Note that the second term increases with the dead-time both 
absolutely and as a fraction of the first term. 
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To determine whether simultaneously recorded motor cortex neurons can be used for real-time 
device control, rats were trained to position a robot arm to obtain water by pressing a lever. 
Mathematical transformations, including neural networks, converted multineuron signals into ‘neu- 
ronal population functions’ that accurately predicted lever trajectory. Next, these functions were 
electronically converted into real-time signals for robot arm control. After switching to this ‘neuroro- 
botic’ mode, 4 of 6 animals (those with >25 task-related neurons) routinely used these brain-derived 
signals to position the robot arm and obtain water. With continued training in neurorobotic mode, 
the animals’ lever movement diminished or stopped. These results suggest a possible means for 


movement restoration in paralysis patients. 


Pioneering studies suggested that motor information in the cor- 
tex is coded through the combined actions of large populations of 
widely tuned neurons rather than by small numbers of highly 
tuned neurons!, Several systems provide further evidence for this 
‘distributed coding’ in brain function23. H owever, such investi- 
gations used serial recordings of single neurons averaged over 
repeated trials, and therefore could not demonstrate neuronal 
population encoding of brain information on a singletrial basis. 
Techniques for simultaneous (parallel) neuronal population 
recording enables surprisingly rich encoding of information in 
the brain using randomly sampled neuronal populations, espe 
cially in the somatosensory“ ¢ and limbic systems’. 

We addressed these issues by simultaneously recording from 
arrays of electrodes chronically implanted in the primary motor 
(M1) cortex and ventrolateral (VL) thalamusin rats trainedina 
forelimb movement task. We asked three questions. First, how 
well can linear or nonlinear mathematical transformations of 
neuronal population activity in the MI cortex and/or VL thala- 
mus encode forelimb movement trajectories? Second, can these 
‘motor codes’ be used to generate an online ‘neuronal popula- 
tion function’ to control a robotic arm in real time with suffi- 
cient accuracy to substitute for animal fordimb movement in the 
trained motor task? Third, might training in this neurorobotic 
mode (rewarding the neural activity itself) change or extinguish 
the previously conditioned movement? 


RESULTS 

Rats were first trained to obtain a water reward by depressing a 
spring-loaded lever to proportionally movea robot arm to a water 
dropper (Fig. 1.). On rdease, water was transferred to the mouth 
passively. Neuronal population recordings in this behavioral mode 
(‘lever-movenent/robot- arm’) were used to derive neuronal pop- 
ulation functions that predicted forelimb movement. A multi- 
channel electronic device then calculated, in real time, the inner 
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product of this neuronal population function (a 32-element 
weight matrix) times integrated spiking activity of 32 simultane 
ously recorded neurons. The output was a single analog voltage 
signal used to position the robot arm. Robot arm control could 
bearbitrarily switched from the lever (actual forelimb movement) 
to this brain-derived neuronal population signal. 

Six rats were trained to control the robot arm by moving a 
lever and were then surgically implanted with recording electrode 
arrays (see Methods). After recovery, populations of 21-46 (mean, 
33.2) single neurons were discriminated from these electrodes 
during lever movement to control arobot arm; from rat 1, we 
simultaneously recorded 46 neurons (32 in MI and 14 in VL; Fig. 
2). The plot depicts averaged responses centered around onset 
of forelimb extension of each neuron and of all MI or VL cells 
over 44 trials. All but 5 neurons modulated firing over this task (= 
99% confidence level; Kolmogorov-Smirnov). In 6 MI-implant- 
ed animals, 16-41 neurons (mean, 33.5) showed significant task 
correlation. 

Movement trials began from rest when the animal reached 
for the lever using antebrachial (forearm) flexion followed by 
brachial (arm) flexion and finally carpal (paw) extension. Just 
above the lever, the forepaw flexed rapidly downward with ante 
brachial extension, touching the lever within a mean of 41 ms. 
Continued extension pressed the lever to position the robot arm 
toward a water drop. The antebrachium was then flexed to release 
the lever, allowing the arm to return to the rat. 

Analysis of repeated trials (using EM G and videos) revealed 
a consistent sequence of forelimb movements. Peri-event his- 
tograms and rank- ordered raster plots were used to characterize 
each neuron’s primary motor correlate in terms of forelimb 
movement elements during these trials. Rank- ordered rasters 
were particularly effective in resolving the part of the observed 
movement sequence best correlated with neuronal activity. Four 
general categories of task-related neurons were observed (Fig. 3). 
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Fig. 1. Experimental protocol. (a) ‘Lever-movement/robot- 
arm' mode: rats were trained to press a lever (b) for a water 
reward; displacement was electronically translated to pro- 
portionally move arobot arm (c) from rest position through 
a slot in barrier (d) to a water dropper (e). The robot 
arm/water drop moved passively to the rest position (to the 
rat). (f) ‘Neuronal-population-function/robot-arm’ mode: 
Rats were chronically implanted with multi-electrode 
recording arrays in the MI cortex and VL thalamus, yielding 
simultaneous recordings of up to 46 discriminated single 
neurons. (g) Superimposed waveforms of 24 such neurons. 
(h) Sample spike trains of two neurons (N1, N2) over 2.0. 
(i) N euronal-population (NP) function extracting the first 
principal component of a 32-neuron population. (j) Switch to 
determine input source (lever movement or NP function) for 
controlling robot-arm position. In experiments, rats typically 
began moving the lever. The input was then switched to the 
NP function, allowing the animal to obtain water through 


articles 


direct neural control of the robot arm. 


‘Pre-flexion’ neurons (13% of task related in M1; 43% 
in VL) discharged before brachial flexion to initiate lever 
reach; ‘flexion’ neurons (16% in MI; 21% in VL) dis- 


charged mainly after the onset of antebrachial flexion; 
‘pre-extension’ neurons (28% in M1; 15% in VL) were 
best correlated with the pre-onset and onset of carpal 
flexion initiating antebrachial extension before forepaw 
placement on the lever, and ‘extension’ neurons (43% 
in M1; 21% in VL) were correlated somewhat with antebrachial 
extension onset and strongly with placement of the forepaw on 
the lever. This last category corresponded closely with previous- 
ly described ‘forepaw placing’ neurons in rat sensorimotor cor- 
tex®. Both pre-extension and extension neurons began to 
discharge ~30-50 ms before forepaw contact. M ost of these dis- 
charged more strongly before touch than during movements 
without contact. M oreover, the timing of these pretouch neural 
response peaks corresponded to timing of sensory gating in neu- 
ronsin SI cortex. These neurons also exhibited ‘motor’ activ- 
ity comparable to the premovement discharges described in the 
MI cortices of many species!3-15 (Fig, 3b). The peri- event his- 
tograms centering around triceps EMG onset in trials in which 
the forepaw was momentarily held motionless on the lever before 
pressing demonstrate that the combined activity of the pre exten- 
sion and extension neurons preceded detectable lever movement 
by as much as 150 ms, and triceps muscle EM G by 100 ms. 
Average amplitude of ensemble activity peaked in the period 
before paw contact with the lever (Fig. 2). These peaks repre- 
sented maximal discharge of the majority of sampled neurons 
(74% in Fig. 2; 71% across all 6 animals) and were therefore pre 
sent in neuronal population (NP) ensemble averages. By selecting 
neurons by peak pre-lever-movement amplitude, it was possible 
to create neuronal population functions composed exclusively of 
neurons (16-32) expressing this tuning function (Figs. 5 and 6). 


Neuronal population coding in spatiotemporal domain 

Because of the heterogeneity of firing characteristics in these neu- 
ronal ensembles, distinct cross-neuron discharge patterns were 
associated with each phase of forelimb movement. Indeed, mul- 
tivariate analysis of variance (MANOVA) yielded significant dif- 
ferences between multineuron discharge patterns measured across 
the four movements in Fig. 2 (F 14,2323 = 12.3, p < 10-9 for this 
case, and p < 10-° for the other animals). Thus conventional sta- 
tistical approaches such as discriminant function analysis 
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(DFA)®721 could be used to mathematically transform popula- 
tion data into ‘neuronal population functions’ that predicted 
each movement on a per-trial basis (82% overall accuracy). 
Though DFA-defined linear neuronal-population functions gen- 
erally predicted movement direction (flexion versus extension), 
they were less able to predict exact timing or displacement. This 
resulted from the neurons’ tendency to phasically discharge 
before onset of movements best associated with their activity, 
suggesting that temporal in addition to spatial mathematical 
transformations in the domains may better predict movement 
from neural activity preceding the movement. 

A two- stage process was developed. First, principal compo- 
nents analysis (PCA) was used to decompose the multineuronal 
covariancein theNP into asmall number of uncorrelated com- 
ponents>617, As all neurons showed activity before lever move- 
ment, the first principal component depicted peaks of NP activity 
preceding lever movement more accurately than the NP ensem- 
ble average in all animals. Simple thresholding (T) of this NP 
function predicted all lever movements in this experiment, and 
87% of lever movements in all animals (Fig. 4b). 

Next, artificial neural networks (ANNs) using dynamic back- 
propagation learning to store temporal information in recur- 
rent connections accurately predicted the lever movement (Fig. 
4d, r = 0.86) by temporally transforming such NP information. 
Limiting input to this AN N to the first principal component was 
sufficient; the second and third components did not improve 
prediction as measured by the mean-squared error loss func- 
tion. Analysis of 44 transformations suggested that the ANN 
used distinct temporal features of the NP signal to predict lever- 
movement timing and magnitude. Lever movement was pre- 
dicted by a steeply sloping pre-lever- movement peak (3 in Fig. 
4b). The remarkably selective ANN also did not respond to activ- 
ity patterns lacking the distinctive shape of this peak (for exam- 
ple, at 1 and 2 in Fig. 4b). Similarly, termination of lever 
movement was correctly predicted within amean 64 ms bya 
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Fig. 2. Color-coded peri-event population response plot 


SE=0.1 shows activity of 46 simultaneously recorded neurons in the 


mmm -2.14 ‘MI cortex and VL thalamus of rat 1, averaged around the 
MM 1.75 onsets of 44 pressing movements. Individual neurons (MI, 32; 
We -1.37 VL, 14) are numbered on the y-axis. Time (x-axis) is mea- 
We -0.99 sured from lever movement. Colors depict instantaneous 
WME -0.61 neuronal firing rates (25-ms bins), standardized for each neu- 
WB 0.23 rons’ peri-event histogram by subtracting the mean and 
(0.14 — dividing by the standard deviation (s.d.). White, vertical lines 
{10.52 indicate mean and black horizontal arrows the ranges of 
i 09 movement onset. Average times of onset of flexion (‘Flex’), 


{1.28 — extension (‘Extens’), ‘Touch-press’, ‘Position’ and ‘Release’ 
(1.66 are indicated by labels above the plot. The blue lines across 
(9) 2.03. the MI and VL population-response plots represent ensem- 
MM 2.41 ble averages of the population activity in these regions over 
M279 44 trials. The ‘pre-LM peak’ immediately precedes onset of 
MM 3.17 lever movement. 
Mm 3.55 

s.d. from 

mean 


distinctively sharp drop in NP activity (4 in Fig, 4b). Similar NP 
properties were found in the other five animals in which MI/VL 
forelimb neurons were recorded. 

Further analysis of such AN Ns suggested that they ‘learned’ 
to make such predictions by encoding distinct tanporal response 
functions into their recurrent circuitry. For example, one trained 


0.001). Finally, these peaks before lever movement were corre- 
lated with extensor (triceps) muscle EM G activity (mean rma, = 
0.64 in 2 animals), beginning amean 26 ms before onset of lever 
movement and continuing for 100-400 ms. As the lever was 
spring loaded, the NP activity in MI and VL preceding lever 
movement was correlated both with the force!31820 and the tra- 


ANN responded to a single test impulse that peaked with 
300-ms latency and subsequently decayed over 1s (inset 
above Fig. 4d). This output timing was remarkably similar 
to the tenporal correlation between the pre-lever-movement 
peak and the lever movement (Fig. 5b). Overall, these find- 
ings suggest that movement-related information may be 
more completely specified by spatiotemporal functions than 
by purely spatial functions. 


Pre-lever-movement peak encodes lever movement 
The strength and specificity of this tight rdationship between 
pre-lever-movement NP activity peaks and lever movements 
was further demonstrated in rank-ordered raster plots and 
correlation analyses (Fig. 5). In trialsin which the pre lever- 
movement peak was small (Fig. 5a, top), theinitial pressing 
movement was insufficient, requiring additional lever pres- 
sure to obtain water. Lever movement was sufficient to posi- 
tion the robot arm under the water dropper within 500 ms 
only in trials in which the pre-lever- movement peak was 
large (Fig. 5a, bottom). 

Time-shifted cross-correlation analysis of the same data 
(Fig. 5b) showed that the NP predicted lever displacement 
over the 200-500 ms following onset of lever movement 
(correlation peak labeled 1; r,,., = 0.76; p < 0.001). By com- 
parison, NP activity following lever movement was nega- 
tively or not correlated with lever movement (2). Moreover, 
the trough at 3 and peak at 4 both indicate prediction of NP 
activity by previous movement. This reversal from a prospec- 
tive (‘motor’) to a retrospective (‘sensory’) mode was asso- 
ciated with the transition from pressing to rdease of the lever, 
when the animal allowed the spring-loaded lever to push the 
forelimb upward. 

Similar temporal correlations were also found in NPs 
containing only MI cortical neurons (Tmax = 0.79; Fax = 0.65 
across the 6 animals) and significantly though more weakly, 
in NP activity of VL thalamic neurons (mean rn, = 0.44, p < 
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Fig. 3. Categories of recorded neurons. (a) Rank-ordered peri-event rasters and 
histograms centered around onset of antebrachial flexion demonstrate motor cor- 
relates of four neuron types. ‘Pre-flexion’ neurons discharged before flexion 
through antebrachial flexion, abruptly terminating on carpal flexion and ante- 
brachial extension. ‘Flexion’ neurons discharged from antebrachial flexion through 
antebrachial extension. ‘Pre-extension’ neurons discharged phasically up to 50 ms 
before carpal flexion. ‘Extension’ neurons discharged phasically and then tonically 
during antebrachial extension in reach-to-touch movements. Vertical scales show 
instantaneous average firing rate. (b) Premovement neural activity during 10 trials 
(in rat 3); rat started lever-press with paw on bar but did not reach. Trials are aver- 
aged around the onset of triceps EMG activity. ‘Triceps’, muscle EMG; ‘N PF’, NP 
activity of 28 MI cortical neurons over the same 10 trials. Mean NP activity led 
EMG onset by 63 ms; bar movement (white arrow) lagged by 33-66 ms. 
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Fig. 4. Comparison of modes of movement ‘coding’ in lever-move- 
ment/robot-arm mode. Vertical dotted lines indicate starts and stops of 
lever movement. (a) Spike train rasters from three neurons showing low, 
middle and high correlation with forelimb movement. (b) Stripchart of 
32-neuron NP function (N PF) binned at 20 ms and smoothed over 2-bin 
windows. Activity peaks relating to overall magnitude of limb movement 
are shown at 1, 2, 3 and 4. W hite, dotted lines mark mean activity during 
the resting period (two-headed arrow). Black, dashed lines mark 3 s.d. 
above mean activity. (c) Vertical position of the lever; threshold ‘drinking’ 
position is at bottom and ‘water drop’ position at top. (d) 
Transformation of N PF (b) with arecurrent neural network (rANN) to 
predict lever movement timing and magnitude as in (c). (e) ‘Impulse 
response’ shows the response of this ANN to an input pulse (arrow) 
equivalent to one spike; impulse-response amplitude is about 1% maxi- 
mum in (d). For both (d) and (e), bin width is 100 ms. Horizontal axes 
are the same for (a-d). C oefficients of correlation (r) with lever position 
in (c) are shown at right for spike trains and N P functions. 


jectory (timing and magnitude) of lever movement. Moreover, 
NP activity preceding lever movement was slightly better corre 
lated with lever displacement than velocity of lever movement 
(measured over 100 ms time bins). Thesame NP activity pre 
ceding lever movement and, after conversion to instantaneous 
velocities, lever displacement in Fig. 5b were used to obtain a 
time shifted correlogram (Fig. 5c). By comparison with Fig. 5b 
(where rma, = 0.76), the maximum correlation between the activ- 
ity peak preceding and velocity after lever movement is highly 
significant, but slightly smaller (1 in Fig. 5¢; rma, = 0.51; p < 
0.001), and is limited to the 200 ms following onset of lever move- 
ment. Similar results were found in the other animals, yielding 
AMEAN Imax = 0.47 for velocity versus 0.65 for displacement. These 
findings may also suggest that movement velocity measured 
200-500 ms after onset was more significantly correlated than 
that measured in the 0-100 msimmediately following movement 
onset with the activity peak before lever movement. This sug- 
gests a complex relationship between the peak amplitude pre 
ceding lever movement and the final position targeted by training. 


Neurorobotic control 

Our ultimate aim was to determine whether the animal could 
use its brain activity, electronically transformed in real-time from 
on-line NP recordings, to control the robot arm with sufficient 
accuracy to obtain water. Because the above analyses demon- 
strated that the NP-function peak amplitude preceding lever 
movement was proportional to lever movement used to position 
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the robot arm, we hypothesized that NP activity itself could be 
used to achieve the same positioning. On the other hand, it was 
not known whether this brief, premovement NP function could 
substitute for the relatively long-duration limb movement with 
which it was normally associated. 

For this purpose, NP functions were generated (using PCA 
as above) for neurons selected for strong activity preceding lever 
movement. To manifest these N Psin real time, a summing ampli- 
fier-integrator circuit allowed each of 32 discriminated spike 
train input channels to be individually weighted and integrated 
into asingle analog voltage output. Control of the robot arm 
could be suddenly switched from the lever-movement/robot-arm 
mode to an NP-function/robot-arm mode (Fig. 1), allowing sub- 
stitution of the NP function for lever movement as the operant 
behavior necessary to obtain the water reward. To maintain the 
normal association between movement of the lever and of the 
robot arm, animals worked in the lever-movement/robot-arm 
mode for about five minutes preceding each experimental ses- 
sion. T his ensured that the animal continued to initiate condi- 
tioned lever movements when the mode was suddenly switched 
to the NP-function/robot-arm mode. These movements provid- 
ed direct timing of the animal’s motor-behavioral ‘intention’ as 
defined by the conditioning task. 

Figure 6b shows typical activity of the NP function over 100 
seconds following this rat’s first switch to NP-function/robot- 
arm mode. In eight of nine lever movements during this period, 
the real-time NP function output successfully moved the robot 
arm to the water-drop position to obtain the water reward (indi- 
cated by asterisks). Over the entire 280-second experiment, the 
animal was 100% efficient (15 of 15 trials) in using NP-function 
activity to retrieve the water reward when appropriately large 
amplitude lever movements were made. In contrast, the N P-func- 
tion failed to reach threshold and the rat did not receive water 
in 8 of 11 trials in which lever movements were incomplete (for 
example, F in Fig. 6b). Two of six animals showed 100% effi- 
ciency during complete lever movements; another 2 showed 76% 
and 91% efficiency on the first try, all of which were sufficient 
to maintain operant behavior. In the renaining two animals, N P- 
function signals were only 67% and 54% efficient in generating 
rewards, insufficient to maintain the behavior for morethan 1-2 
minutes. 

The success of the NP function in maintaining conditioned 
behavior depended on the strength of activity preceding lever 
movement relative to its background variability”!, This was most 
importantly related to the number and selectivity of the neurons 
whose activity was integrated to generate the NP function. In Fig. 
6b, for example, the N P-function threshold for moving the robot 
arm to the water dropper was 10.25 s.d. above the mean, but 
among the 32 neurons used to encode it, mean peak activity pre 
ceding lever movement was on average 3.2 s.d. above the mean. 
Thus, the NP function resolved lever movement much more clear- 
ly than any single constituent neuron. Across the 4 successful ani- 
mals, the activity peaks preceding lever movement were on average 
9.2 s.d. above the mean (range, 7.64- 10.25), but only 3.6 for the 
individual neurons. In the 2 unsuccessful rats, the yield of task- 
related neurons was relatively low (16 and 19), giving relatively 
weaker activity peaks preceding lever movement (5.2 and 5.7 s.d. 
above the mean). Thus, an activity peak of 6-8 s.d. above the 
mean and an NP of 20-25 task-related neurons were required to 
successfully control the robot arm and maintain conditioning. 

These results showed that animals could substitute the NP 
function for limb movement despite its tendency to lead lever 
movement, triggering delivery of water before movement onset. 
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Fig. 5. Premovement N P activity predicts lever movement. (a) 
Rank-ordered peri-event rasters show NP-function activity 
(N PF; as in Fig. 4), left, and lever movement (LM), right, over the 
same 74 trials. Trials are rank ordered (from bottom) by peak 
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Therefore, this provided a test of whether the animal could adapt 
its operant behavior and/or its NP activity to the changed 
response criterion. These experiments were typically begun with 
a short period of lever-movement/robot-arm mode testing to 
confirm the normal correlation between the activity peak pre 
ceding lever movement and its initial trajectory (Fig. 6c, left). 
After switching to N P-function/robot-arm mode (Fig. 6c, right), 
animals typically continued to press the lever down to the origi- 
nal threshold position for obtaining the water for several trials, 
presumably reflecting the high correlation between lever move- 
ment and the preceding activity peak (Fig. 5b). During subse 
quent trials, however, this normally high correlation declined 
(from r = 0.81, p = 0.004 over the first 10 trials in Fig. 6c, right, to 
insignificant correlation for all subsequent sets of 10 trials). Even 
though the animal continued to make sporadic lever movements, 
usually following failures (for instance, trials 30 and 45 follow- 
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(1.5 s). Traces at top and right show averaged NP function and 
lever movement. (c) Correlogram similar to (b) correlating N P 
activity with lever velocity rather than displacement. 


ing 29 and 44), the animal was generally able to obtain 
water without moving the lever to the threshold rou- 
tinely attained under the lever-movement/robot- arm 
mode. Indeed, 11 of these trials were rewarded without 
any lever movement, and in one trial, without even a 
forelimb-reaching movement, the animal simply resting 
its paw on the lever. 

Similar results were obtained in all four animals able 
to work in the NP-function/robot-arm mode. By the sec- 
ond day of this conditioning, they all maintained a 
60-100% success rate, even when the normally high cor- 
relation between lever movement and preceding activi- 
ty peak (Tmean = 0.65) fell to insignificant levels (Tmean = 
0.15). Despite these changes, analysis of the magnitude 
and timing of the first five principal components of the 
NP-function activity revealed no significant changes in 
the overall distribution of neuronal discharges. Thus, it 
is unlikely that the observed reduction in movement 
magnitude could be explained by a selective reduction 
of activity in a certain subclass of neuronsin this popu- 
lation, in either the MI or theVL. 


DIscussION 

This investigation demonstrated that information 
extracted from simultaneously recorded populations of 
single neurons in the brain can be used to proportion- 
ally drive an external motion device in real time This 
was made possible by the recent availability of techniques for 
multisite, multineuron recording, and provides further evidence 
of the importance of activity across neuronal populations for 
information processing in the brain. The current results also cor- 
roborate previous suggestions?2-4 that neuronal populations sur- 
pass single neurons in the ability to smoothly encode brain 
information pertaining to asensory image or amovement. Here 
this hypothesis was directly tested by requiring animals to use 
such neuronal population activity to control the robot arm to 
obtain water. 

Another important finding was that representations of the 
spatiotemporal domain of neural information (using recurrent 
ANNs, for instance) predicted movement timing with greater 
accuracy than representions of the spatial domain alone. In par- 
ticular, a short but highly synchronized period of neural activity 
(the pre-lever-movement peak) was a better predictor of lever 
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Fig. 6. N eurorobotic mode. (a) Spike 
trains from three neurons for 100 seconds 
after switching to N P-function/robot-arm 
(N PF/RA) mode. (b) NP function for the 
same period used to electronically drive a b 
robot arm in realtime. Asterisks indicate 
pre-movement N P peaks in trials in which 

the robot arm was successfully moved to 

the water drop ‘T (NPF)’ using this real 

time NP signal. (c) N PF amplitude in the 0 
100 ms before lever movement shows loss 

of correlation between the N PF and lever 
movement (LM) during continued training © 
in NPF/RA mode. On day 2, the animal 
was first reconditioned in the LM/RA for 
27 trials (left) followed by 50 trials in 
N PF/RA mode (right). Correlation coeffi- 
cients (r) are indicated by thick horizontal 
lines over included trials. Significant corre- 
lation (p = 0.004) was found only in the 
first ten N PF/RA trials. ‘T (LM)’ indicates 
the threshold position and ‘T(N PF)’ indi- 
cates activity threshold for reward in 
LM/RA and NPF/RA modes, respectively. 
Animals failing to attain the T(LM) thresh- 
old in 500 ms of LM/RA often reached it 
with additional pressing. 
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movement than was neural activity recorded during the lever 
movement itself. This is consistent with our finding that distinct 
temporal patterns of multi-neuronal ensemble discharge in the 
monkey somatosensory cortex encode specific parameters of sen- 
sory stimulation2. 

Wealso demonstrated that the pre lever- movement peak’s 
normal ability to predict movement can be quickly modified by 
training in the neurorobotic mode. Thus, even though the pre. 
lever-movement peak normally shows ‘movement coding’ (as 
defined by the usual correlational approach), this does not guar- 
antee an obligatory relationship with movement. Though this 
adaptability implies a level of abstraction normally ascribed to 
the premotor cortices, thereis evidence that the MI shows simi- 
lar functional adaptability®. Moreover, our findings are directly 
consistent with classic reports that MI cortical single units can 
be dissociated from movement by conditioning?” 34. These results 
are also generally consistent with reports of remapping of motor 
output functions of the M1 cortex after peripheral injury2. The 
current results also show that the NP function itself could rapid- 
ly replace overt movement as the operant behavior for this con- 
ditioning protocol. On the other hand, the context of voluntary 
movement was not extinguished, as our animals generally con- 
tinued to reach to the lever, but not to press it. It seems, there- 
fore, that this decorrelation of motor cortex neural activity from 
movement was facilitated by maintenance of the overall motor 
context. It is thus conceivable that the animals could have formed 
an internal representation of the intended movements and then 
aborted their expression. 

The feasibility of using simultaneously recorded N Ps to con- 
trol external movement devices demonstrated by this study rais- 
es the possibility that paralyzed patients could use such recordings 
to control external devices or even their own muscles through 
functional electrical stimulation3-35, Considering the require 
ment for such patients to ‘relearn’ neural control of movement, it 
is notable that the current studies demonstrate modifiability of 
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cortical NP codes. Finally, beyond this clinical application, the 
general strategy developed here of using brain-derived signals to 
control external devices may provide a unique new tool for inves- 
tigating information processing within particular brain regions. 


METHOoDs 

Implantation. M ethods for surgery, multineuron recording and analy- 
sis have been published? ®111617, Briefly, six Long-Evans (hooded) rats 
were chronically implanted with arrays or bundles of microwire elec- 
trodes in the M| cortex and VL thalamus. Arrays consisted of two par- 
allel rows of eight electrodes (2.0-mm long, rows separated by 0.5). Arrays 
were implanted rostrocaudally across layer V of the primary motor cor- 
tex (M1) forelimb area*6; location was verified by microstimulation to 
produce forelimb movements at threshold (80-200 wA)}3, characteriza- 
tion of motor correlates (see Fig. 3) and histology. All M| cortical neu- 
rons described were ‘forelimb-positive’ by these criteria. Bundles were 
implanted in the VL thalamus within a ~1.0-mm radius. Two rats were 
also implanted with four bipolar EM G electrodes (twisted pairs of seven- 
stranded stainless steel, Medwire, Mt. Vernon, New York) in forelimb 
muscles, including the triceps and biceps long head and extensor and 
flexor carpi/digitorum muscles. 


Signal processing. Approximately one week after surgery, animals were 
placed in the behavior chamber, and their head-mounted chronic 
implants were connected via a wire harness to a 64-channel M any-Neu- 
ron-Acquisition-Processor (M NAP; Plexon, Dallas, Texas). Amplified 
and filtered waveforms from one to two single neurons (at least 5x noise) 
per channel were continuously discriminated using on-screen manipu- 
ation of multiple time voltage windows. Times of single neuron (and 
also multi-motor-unit) action potentials were stored on aM otorolaVME 
162 computer, synchronized with analog recordings of robot-arm posi- 
tion and videotape recordings of the animals’ motor behavior. Discrim- 
inated waveforms (40 kHz resolution; Fig. 1) could be uploaded into a 
PC computer for off-line graphical display and analysis. Timing pulses 
for each of 32 discriminated action potentials were also routed via sepa- 
rate channels into a real-time 32-channel, spike-integrator board to gen- 
erate a single analog signal to control robot arm position. 

Both single neuron and neuronal population function signals were 
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analyzed using Stranger (Biographics, Winston-Salem, North Carolina) 
and NEX (Plexon, Dallas, Texas) to generate stripcharts, peri-event his- 
tograms (PEHs), rasters and correlograms. Significant events for such 
analysis were obtained from the lever output, field- by-field analysis of 
videotapes (17-ms resolution, system from Lafayette Instruments, 
Lafayette, Indiana) and EMG recordings. PEHs were used to initially 
characterize single: neuron movement correlates. To visualize the statis- 
tical dispersion in such PEHs, bins were standardized by subtracting the 
mean and dividing by the standard deviation (Fig. 2), allowing one con- 
fidence interval to be calculated for all histogram bins. The Kolmogorov- 
Smirnov one-sample test was also used as anon-parametric, robust and 
relatively assumption-free statistical method for evaluating PEH results. 
Significance among multiple neurons was assessed using a multivariate 
analysis of variance (MANOVA), typically with a repeated measures, sin- 
gle-factor design across non-overlapping experimental groups. Dis- 
criminant function analysis (DFA)®7 was performed by randomly 
dividing the data from the neuronal population into a learning set and 
a test set (Statistica, StatSoft, Tulsa, Oklahoma). 

Two techniques were used to transform the timing pulses for the dis- 
criminated action potentials of simultaneously recorded neurons into 
neuronal population functions. Principal components analysis (PCA) 
was used to generate linear neuronal population functions based on 
recurring patterns of covariance among the neuronal activity as previ- 
ously described>2!22, PCA provides weighting of each neuron’s contri- 
bution to the population average according to its patterns of correlation 
with other neurons, thereby concentrating the salient ‘signal’ embedded 
within the neural activity into a small number of orthogonal principal 
components and effectively separating them from the stochastic activity 
of individual neurons. PCA involved eigenvalue rotation of the correla- 
tion matrix between firing rates in equivalent time bins of recorded neu- 
rons. This produced a set of principal components (eigenvectors) that 
explained successively smaller amounts of the total variance. Compo- 
nents could be reconstructed as a weighted population average of the 
recorded neurons using the rotational coefficients as weights. 


Spatio-temporal transformations using recurrent artificial neural net- 
works (ANNs). The ANNs were built using the ‘Neurosolutions’ pack- 
age (Neural Dimensions, Gainesville, Florida). Dynamic 
back- propagation learning (using the learning data set) was used to adjust 
the weightings within this network to optimally transform the NP input 
(from the test data set) into an output function that closely matched the 
lever-movement predictor function (Fig. 4d). Testing was carried out 
only after the mean- squared error of the prediction (loss function) 
asymptotically reached aminimum level. 
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Data analysis 


A neuron was considered selective to a stimulus group if: (1) the firing rate during stimulus 
presentation was different from the preceding baseline (Wilcoxon test, < 0.05), (2) an 
analysis of variance and pairwise comparisons (Wilcoxon test) addressing whether there 
were differences among the stimulus groups yielded P < 0.05 and (3) an ANOVA 
(parametric and non-parametric) comparing the variability to distinct stimuli within the 
selective category to the variability to repeated presentations of the same stimulus showed 
P>0.05. We observed neurons selective to faces, objects, spatial layouts and other stimuli. 

If the across-groups comparisons were not significant but the activity was different from 
baseline, the neuron was defined as responsive but non-selective. To take into account any 
effects due to the different intervals we also compared the responses in a 600-ms window 
centred on the peak firing rate. The peak, latency and duration were estimated from the 
spike density function”. For the selective neurons we computed the probability of error, 
P,, for classifying the stimulus as belonging to the preferred stimulus category or not'*”’. 
We did not observe any difference between the right and left hemisphere neurons. 
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Signals derived from the rat motor cortex can be used for 
controlling one-dimensional movements of a robot arm’. It 
remains unknown, however, whether real-time processing of 
cortical signals can be employed to reproduce, in a robotic 
device, the kind of complex arm movements used by primates 
to reach objects in space. Here we recorded the simultaneous 
activity of large populations of neurons, distributed in the pre- 
motor, primary motor and posterior parietal cortical areas, as 
non-human primates performed two distinct motor tasks. Accu- 
rate real-time predictions of one- and three-dimensional arm 
movement trajectories were obtained by applying both linear and 
nonlinear algorithms to cortical neuronal ensemble activity 
recorded from each animal. In addition, cortically derived signals 
were successfully used for real-time control of robotic devices, 
both locally and through the Internet. These results suggest that 
long-term control of complex prosthetic robot arm movements 
can be achieved by simple real-time transformations of neuronal 
population signals derived from multiple cortical areas in 
primates. 

Several interconnected cortical areas in the frontal and parietal 
lobes are involved in the selection of motor commands for produc- 
ing reaching movements in primates” *. The involvement of these 
areas in many aspects of motor control has been documented 
extensively by serial single-neuron recordings of primate 
behaviour™*”, and evidence for distributed representations of 
motor information has been found in most of these studies!-', 
but little is known about how these cortical areas collectively 
influence the generation of arm movements in real time. The 
advent of multi-site neural ensemble recordings in primates" has 
allowed simultaneous monitoring of the activity of large popula- 
tions of neurons, distributed across multiple cortical areas, as 
animals are trained in motor tasks'’. Here we used this technique 
to investigate whether real-time transformations of signals gener- 
ated by populations of single cortical neurons can be used to mimic 
in a robotic device the complex arm movements used by primates to 
reach for objects in space. 

Microwire arrays were implanted in multiple cortical areas of two 
owl monkeys (Aotus trivirgatus)'*"'’. In the first monkey, 96 micro- 
wires were implanted in the left dorsal premotor cortex (PMd, 16 
wires), left primary motor cortex (MI, 16 wires)’”"’, left posterior 
parietal cortex (PP, 16 wires), right PMd and MI (32 wires), and 
right PP cortex (16 wires). In the second monkey, 32 microwires 
were implanted in the left PMd (16 wires) and in the left MI (16 
wires). Recordings of cortical neural ensembles began 1—2 weeks 
after the implantation surgery and continued for 12 months in 
monkey 1, and 24 months in monkey 2. During this period, the 
monkeys were trained in two distinct motor tasks. In task 1, animals 
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Figure 1 Experimental design. a, Schematic diagram depicting the experimental 
apparatus employed to use simultaneously recorded cortical ensemble data from primate 
behaviour to control the movements of local and remote robotic devices. In this design, 
both linear and ANN models are continuously updated during the recording session. 
Control of a local device is achieved through a local area network (LAN), while remote 


— 


1 second 


control requires transmission of brain-derived signals through the Internet. b, c, 
Simultaneously recorded neuronal activity in five cortical areas in monkey 1 (b) and two 
cortical areas in monkey 2 (c) during the execution of 1D movements. PP, posterior 
parietal cortex; M1, primary motor cortex; PMd, dorsal premotor cortex; iM1/PMd, 
ipsilateral M1 and PMd. 
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Figure 2 Real-time control of 1D hand movements. a, b, Coherence analysis reveals that 


most cortical neurons are significantly coupled with different frequency components of the 
movements. c, d, Observed (black) and real-time predicted 1D hand movements using 
linear (red) and ANN (blue) models in monkey 1 (¢) and 2 (d). e, f, Correlation coefficient 
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variation for predicted hand movements, using linear (red) and ANN (black) models, in one 
recording session in monkey 1 (e) and 2 (f). g, Real-time 1D movements of a local (blue) 
and remote (red) robot arm obtained in monkey 1 by using the linear model. 
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made one-dimensional hand movements to displace a manipulan- 
dum in one of two directions (left or right) following a visual cue. In 
task 2, the monkeys made three-dimensional hand movements to 
reach for small pieces of food randomly placed at four different 
positions on a tray. Cortical recordings were obtained while the two 
subjects were trained and tested on both tasks (Fig. 1a). 

Figure 1b and c illustrate samples of the raw neuronal data 
obtained while the animals performed task 1. In both monkeys, 
coherence analysis'’”'revealed that the activity of most single 
neurons from each of the simultaneously recorded cortical areas 
was significantly correlated with both one-dimensional (Fig. 2a and 
b) and three-dimensional hand trajectories, although the degree 
and frequency range of these correlations varied considerably 
within and between cortical areas. We then investigated whether 
both linear’?! and artificial neural network (ANN)”” algorithms 
could be used to predict hand position in real time. For one- 
dimensional movements, we observed that both algorithms yielded 
highly significant real-time predictions in both monkeys (Fig. 2c 
and d). These results were obtained in spite of the fact that the 
trajectories were quite complex, involving different starting posi- 
tions, as well as movements at different velocities. For example, in 
the session represented in Fig. 2c, the activity of 27 PMd, 26 MI, 28 
PP, and 19 ipsilateral MI/PMd neurons in monkey 1 allowed us to 
achieve an average correlation coefficient of 0.61 between the 
observed and predicted hand position (60-minute session, range 
0.50—0.71, linear model; 0.45-0.73, ANN; P <0.001'”'). Figure 
2d illustrates similar real-time results obtained by using a smaller 
sample of neurons (8 PMd and 27 MI) in monkey 2 (average 
r = 0.72, range 0.47-0.79, linear model; average r = 0.66, range 
0.42-0.71, ANN, P < 0.001). No large differences in fitting accuracy 
were observed between linear and ANN algorithms in either animal 
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(Fig. 2c and d, linear prediction shown as green line, ANN as red 
line). As shown in Fig. 2e (monkey 1) and Fig. 2f (monkey 2), the 
performance of both algorithms improved in the first few minutes 
of recordings and then reached an asymptotic level that was 
maintained throughout the experiment. In both monkeys, highly 
significant predictions of hand movement trajectories were 
obtained for several months. 

To reduce the influence of dynamic changes in the coupling 
between neuronal activity and movements and other non-station- 
ary influences in our real-time predictions, both linear and ANN 
models were continuously updated throughout the recording ses- 
sions. This approach significantly improved the prediction of hand 
trajectories. For example, when predicting the last 10 minutes of 
50-100-minute sessions, the adaptive algorithm performed 55% 
(20 sessions, median) better than a fixed model based on the initial 
10 minutes, or 20% better than a model based on the 30—40-minute 
segment of the session. 

Because accurate hand trajectory predictions were achieved 
early on in each recording session and remained stable for long 
periods of time, we were able to use brain-derived signals to 
control the movements of robotic devices (Phantom, SensAble 
Technologies)” in real time (Fig. 2g). In addition, we were able to 
broadcast these motor control signals to multiple computer 
clients by using a regular Internet communication protocol 
(TCP/IP, Fig 1a) and control two distinct robots simultaneously: 
one at Duke University (Fig. 2g, blue line) and one at MIT (Fig. 2g, 
red line). 

Next, we investigated whether the same cortical ensemble activity 
and models could be used to predict the complex sequences of 
three-dimensional hand movements used by primates in a food- 
reaching task (task 2). These movements involved four phases: 
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Figure 3 Real-time prediction of 3D hand movements. a, b, 3D hand movement 
trajectories produced by monkey 1 (a) and 2 (b) during single experimental sessions. 
c, schematic diagram of the four possible target locations in the food reaching task. 
d, e, samples of observed (black line) and real-time predicted (red line) 3D hand 
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movement for monkey 1 (d) and 2 (e). f, g, Correlation coefficient variation for x (black 
line), y (blue line) and z (red line) dimensions of predicted 3D hand movements using the 
linear model. 
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reaching for the food, grasping the food, bringing the food to the 
mouth and returning to the start position (Fig. 3a and b). Because 
these animals were not overtrained, their movement trajectories 
were highly variable. For example, in the session represented in Fig. 
3a (monkey 1) the dispersion of hand trajectories was 7.0 by 7.5 by 
6.0 cm (or 315 cm’); in Fig. 3b (monkey 2) the dispersion was even 
bigger, 11.5 by 10.5 by 10.0cm (or 1,207.5cm’). Nonetheless, in 
both animals, the same linear and ANN models described above 
provided accurate predictions of three-dimensional hand trajec- 
tories in four different directions during 25—60-minute experimen- 
tal sessions (60-208 trials). Figures 3d and e illustrate several 
examples of observed (black lines) and predicted (red lines) 
sequences of three-dimensional movements produced by monkey 
1 (Fig. 3d) and 2 (Fig. 3e). After initial improvements, the predic- 
tions of three-dimensional hand trajectories reached asymptotic 
levels that were maintained throughout the experiments (Fig. 3f and 
g). The three-dimensional predictions were comparable to those 
obtained for one-dimensional movements (monkey 1: r = 0.74, 
0.72 and 0.56 for the x-, y- and z-dimensions, respectively; monkey 
2: r = 0.70, 0.54 and 0.77; 20-minute averages). 

Further demonstration of the robustness of our real-time 
approach was obtained by investigating how well model parameters 
obtained for one set of hand movements could be used to predict 
hand trajectories to other directions. For example, by training our 
linear model only with hand movements directed to targets on the 
right (targets 1 and 2) we were able to predict accurately hand 
trajectories to targets on the left (targets 3 and 4). The same was true 
for the reverse case, that is, using parameters derived from left 
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Figure 4 Neuron-dropping analysis. a, b, Neuron-dropping curves (thick lines) obtained 
in a single session for all cortical neurons in monkey 1 (a) and 2 (b) are precisely fitted by 
hyperbolic functions (thin lines). c, d, Neuron-dropping curves (thick lines) and 
corresponding hyperbolic functions (thin lines) for each cortical area for monkey 1 (c) and 
2 (d). e, Range of fitted hyperbolic functions, xis the number of neurons and cis the fitted 
constant. f, g, Estimated number of neurons per cortical area required to reach an F° of 
0.9 using the hyperbolic functions in monkey 1 (f) and 2 (g). Lower values represent 
higher mean contribution per neuron. Three asterisks, P< 0.001. 
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movements to predict movements to the right (monkey 1, r = 0.80, 
0.70 and 0.67 for the x-, y- and z- dimensions; monkey 2, r = 0.68, 
0.53 and 0.81; averages for both conditions). Predictions of distal 
(targets 2 and 4) movements by training the model only with 
proximal (targets 3 and 1) hand trajectories and vice versa were 
comparably accurate (monkey 1, r= 0.81, 0.71 and 0.74 for the x-, y- 
and z-dimensions; monkey 2, r = 0.69, 0.63 and 0.79; averages). 

We next analysed how each of the 2—4 different cortical areas 
contributed to the prediction of one-dimensional movements by 
calculating the average effect of removing individual neurons, one at 
a time, from the neuronal population used in each real-time session. 
This neuron-dropping analysis was carried out independently for 
each of the cortical areas, as well as for the combination of all of 
them. We found that hyperbolic functions could fit (r range 0.996- 
0.9996) all curves that resulted from the neuron-dropping analysis, 
using both the linear and ANN models in both animals. Figure 4a—e 
illustrates typical neuron-dropping curves and the corresponding 
hyperbolic fits. Extrapolations of the hyperbolic curves (Fig. 4f) 
revealed that 90% correct real-time prediction of one-dimensional 
movements could be theoretically achieved by applying the linear 
model to either 480 + 65.7 PMd neurons, 666 + 83.0 M1, 629 + 
64.2 PP or 1,195 + 142 ipsilateral MI/PMd neurons in monkey 1 
(average and s.e.m., 10 sessions). In monkey 2, the same level of 
accuracy would require either 376 + 42.3 PMd neurons or 869 + 
127.4 MI neurons (Fig. 4g). Thus, in both monkeys significantly 
fewer PMd (red) neurons would theoretically be required to achieve 
the same level (90%) of one-dimensional hand movement predic- 
tion accuracy (P < 0.001, Wilcoxon), that is, on average, PMd 
neurons provided the highest contribution to the predictions. MI 
(light blue) and PP (dark blue) ensembles provided comparably 
lower contributions, whereas neurons located in the ipsilateral MI 
cortex accounted for the lowest amount of variance (yellow line). 
When all recorded cortical neurons were combined, the extrapola- 
tion of the hyperbolic functions produced identical theoretical 
estimates for 90% prediction accuracy in both monkeys (monkey 
1, 625 + 64 neurons; monkey 2, 619 + 73 neurons, average and 
s.e.m.; P = 0.70, Wilcoxon, not significant). 

These results are consistent with the hypothesis that motor 
control signals for arm movements appear concurrently in large 
areas of the frontal and parietal cortices”®, and that, in theory, each 
of these cortical areas individually could be used to generate hand 
trajectory signals in real time. However, the differences in estimated 
neuronal sample required to predict hand trajectories using a single 
cortical area probably reflect the functional specializations of these 
regions. Thus, the differences observed here are consistent with 
previous observations that PP and MI activity are influenced by 
motor parameters other than hand position (for example, visual 
information in the case of PP*”, or information related to the 
motor periphery in the case of MI**”’). The relative contributions of 
these cortical areas may possibly also change according to such 
factors as the demands of the particular motor task, training level or 
previous motor experience””’. 

In conclusion, we demonstrated that simultaneously recorded 
neural ensemble activity, derived from multiple cortical areas, can 
be used for the generation of both one-dimensional and three- 
dimensional signals to control robot movements in real time. 
Contrary to previous off-line algorithms’, our real-time 
approach did not make any a priori assumptions about either the 
physiological properties (for example, shape of the tuning curve) of 
the single neurons, or the homogeneity of the neuronal population 
sample. Instead, by using random samples of cortical neurons, we 
obtained accurate real-time predictions of both one-dimensional 
and three-dimensional arm movements. In this context, our find- 
ings support the notion that motor signals derived from ensembles 
of cortical neurons could be used for long-term control of prosthetic 
limb movements*”’. We have shown that chronically implanted 
microwires can yield reliable recordings in primates for at least 24 
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months; this suggests that a combination of denser multi-wire 
arrays with implantable integrated circuits, designed to handle all 
real-time signal processing and mathematical analysis, could one 
day form the basis of a brain—machine interface for allowing 
paralysed patients to control voluntarily the movements of 


prosthetic limbs”. 


Methods 


For more detailed methods see Supplementary Information. 


Electrophysiological recordings and behavioural tasks 


Multiple microelectrode arrays (NBLABS, Dennison) containing 16-32 50-m diameter 
microwires were chronically implanted in different cortical areas'*"'° in two owl 
monkeys. Stereotaxic coordinates, published microstimulation maps'”'* and intra- 
operative neural mapping were used to locate the premotor, primary motor and 
posterior parietal cortical areas. A many-neuron acquisition processor (MNAP, Plexon) 
was used to acquire and distinguish activity from single neurons'®. Both monkeys were 
trained and tested on two behavioural tasks. In the first task, the subjects were trained to 
centre a manipulandum for a variable time and then to move it either to left or right 
targets in response to a visual cue in order to receive a juice reward. The position of the 
manipulandum was recorded continuously with a potentiometer. In the second task, the 
monkeys were trained to place their right hands on a small platform, located waist high 
and next to their bodies. When an opaque barrier was lifted, the monkeys reached out 
and grabbed a small piece of fruit from one of four fixed target locations on a tray 
mounted in front of the platform. In both tasks, the location and orientation of the wrist 
in three-dimensional space was continuously recorded using a plastic strip containing 
multiple fibre optic sensors (Shape Tape, Measurand) attached to the right arms of the 
monkeys. Analogue signals from the strip were sampled at 200 Hz. All sessions were also 
videotaped. 


Data analysis 


Predictions of hand position based on simultaneous neural ensemble firing were obtained 
by applying both a linear model and ANNs to these data. In the linear model, the 
relationship between the neuronal discharges in the matrix X(t), and hand position (1D or 
3D) in Y(t) is 


Y(t) =b+ s a(u)X(t — u) + €(t) 


u=—m 


where b are the Y-intercepts in the regression, and a is a set of impulse response functions 
describing the weights required for the fitting as function of time lag u. €(t) are the residual 
errors. Hence, in this model the series in X (that is, neuronal firing in time) are convolved 
with the weight functions a so that the sum of these convolutions plus b approximates Y 
(hand trajectory)’. Offline analysis was first performed to test the validity of this model. 
Y-intercepts and impulse response functions with a resolution of 5 ms were calculated 
using a frequency-domain approach"””’. Real-time prediction was achieved by using a 
slightly modified linear model. Neural data representing potential feedback information 
from movements were not included in the model. Each neuron’s discharges were counted 
in 100-ms bins. Y-intercepts and weights were calculated using standard linear regression 
techniques”. The performance of this simplified real-time model was only very slightly 
inferior to the complete model outlined above. 

Our strategy for applying ANNs is described elsewhere'*”’. Here, we employed the same 
data structure described for the linear model. Several feed-forward ANNs were evaluated 
offline. The best results were obtained using one hidden layer with 15-20 units, linear 
output units, the Powell-Beale conjugate gradient training algorithm”, and an early 
stopping rule. Predictions obtained with ANNs were comparable to those for the linear 
algorithm. 

During real-time prediction of hand position, up to 10 minutes of data were first 
collected to fit the linear and ANN models. The resulting models were then sent back to the 
data acquisition computer for instantaneous and continuous prediction of hand position 
using the neuronal ensemble data acquired in real time. During the remainder of the 
experimental session, both models continued to be calculated repeatedly, using the most 
recently recorded 10 minutes of data. All calculations were performed by software 
designed in MATLAB (The Mathworks, Natick, Massachusetts, USA). 


Real-time control of robot arms 


Predictions of hand positions were broadcasted by a TCP/IP server to one or more 
computer clients. One of these clients controlled the movements of a robot arm (Phantom 
model A1.0; Phantom, SensAble Technologies)” locally at Duke University, and another 
client was used to control simultaneously the movements of a remote robot arm (Phantom 
DeskTop) at MIT using the Internet. Signals describing the position of the robot arm in 
space were recorded on each client machine. 


Neuron-dropping analysis 


The relation between ensemble size and the goodness of fit for our models was analysed 
offline by randomly removing single neurons, one at a time, and fitting the models again 
using the remaining population until only one neuron was left. This procedure was 

repeated 25-30 times for each ensemble, so that an average curve was obtained to describe 
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R’ asa function of ensemble size. Neuron-dropping curves were obtained for all cortical 
areas (separately or combined) in each monkey, and simple hyperbolic functions were 
used to fit them with very high accuracy. Next, we used these functions to estimate the 
ensemble size that would theoretically be required to attain a R? of 0.9. 
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Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. 
Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a 
closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor 
parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical 
activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor 
parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc 
operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using 
visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not 
move. Learning to operate the BMlIc was paralleled by functional reorganization in multiple cortical areas, suggesting 


that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations. 


Introduction 


Traumatic lesions of the central nervous system as well as 
neurodegenerative disorders continue to inflict devastating, 
and so far irreparable, motor deficits in large numbers of 
patients. Every year, spinal cord injuries alone are responsible 
for the occurrence of about 11,000 new cases of permanent 
paralysis in the United States (Nobunaga et al. 1999). These 
cases add up to an already sizeable population of patients, 
estimated at 200,000 in the United States (Nobunaga et al. 
1999), who have to cope with partial (as in the case of 
paraplegics) or almost total (i.e., quadriplegia) body paralysis. 

Until very recently, the main thrust of basic research on 
restoration of motor functions after spinal cord injuries 
focused on reconstructing the connectivity and functionality 
of damaged nerve fibers (Ramon-Cueto et al. 1998; Uchida et 
al. 2000; Bomze et al. 2001; Bunge 2001; Schwab 2002). While 
this repair strategy has produced encouraging results, such as 
limited restoration of limb mobility in animals, the goal of 
restoring complex motor behaviors, such as reaching and 
grasping, remains a major challenge. 

Two decades ago, an alternative method for restoring 
motor behaviors in severely paralyzed patients was proposed 
(Schmidt 1980). This approach contends that direct interfaces 
between spared cortical or subcortical motor centers and 
artificial actuators could be employed to “bypass” spinal cord 
injuries so that paralyzed patients could enact their voluntary 
motor intentions. Initial experimental support for a corti- 
cally driven bypass came from the studies conducted by Fetz 
and collaborators (Fetz 1969; Fetz and Finocchio 1971, 1975; 
Fetz and Baker 1973), who demonstrated that macaque 
monkeys could learn to selectively adjust the firing rate of 
individual cortical neurons to attain a particular level of cell 
activity if provided with sensory feedback that signaled the 
level of neuronal firing. 

Recent studies in rodents (Chapin et al. 1999; Talwar et al. 
2002), primates (Wessberg et al. 2000; Serruya et al. 2002; 
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Taylor et al. 2002), and human subjects (Birbaumer 1999) 
have rekindled interest in using brain-machine interfaces 
(BMIs) as a potential alternative for spinal cord rehabilita- 
tion. These experiments have demonstrated that animals can 
learn to utilize their brain activity to control the displace- 
ments of computer cursors (Serruya et al. 2002; Taylor et al. 
2002) or one-dimensional (1D) to three-dimensional (3D) 
movements of simple and elaborate robot arms (Chapin et al. 
1999; Wessberg et al. 2000). 

Despite these initial results, several fundamental issues 
regarding the operation of BMIs, ranging from basic electro- 
physiological issues to multiple engineering bottlenecks, 
remain a matter of considerable debate (Nicolelis 2001, 
2003; Donoghue 2002). For example, although most agree that 
a BMI designed to reproduce arm/hand movements will 
require long-term and stable recordings from cortical (or 
subcortical) neurons through chronically implanted electrode 
arrays (Nicolelis 2001, 2003; Donoghue 2002), there is 
considerable disagreement on what type of brain signal 
(single unit, multiunit, or field potentials [Pesaran et al. 2002]) 
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would be the optimal input for a such a device. Research 
groups that propose the use of single-unit activity also 
diverge on the assessments of whether small (eight to thirty) 
(Serruya et al. 2002; Taylor et al. 2002) or substantially larger 
(hundreds to thousands) numbers of single units may be 
necessary to operate a BMI efficiently for many years 
(Wessberg et al. 2000; Nicolelis 2001). 

The issues of what signal to use and how much neuronal 
tissue to sample are linked to the question of what type of 
motor commands may be extracted from brain activity. Up to 
now, animal studies have demonstrated the capability of 
extracting a single motor parameter (e.g., hand trajectory, 
position, direction of movement) from brain activity in order 
to operate a BMI (Wessberg et al. 2000; Taylor et al. 2002). 
While it is well known that cortical neuronal activity can 
encode a variety of motor parameters (Fetz 1992; Messier and 
Kalaska 2000; Johnson 2001), it is not clear which cortical 
areas would provide the best input for a BMI designed to 
restore multiple features of upper-limb function. A couple of 
laboratories have focused on the primary motor cortex (M1) 
(Serruya et al. 2002; Taylor et al. 2002), while another group 
has selected the parietal cortex as the main input to a BMI 
(Pesaran et al. 2002). Our previous studies have suggested 
that, because of the distributed nature of motor planning in 
the brain, neuronal samples from multiple frontal and 
parietal cortical areas ought to be employed to operate such 
devices (Wessberg et al. 2000; Nicolelis 2001, 2003; Donoghue 
2002). Another important issue that has received little 
attention is how the interposition of an artificial actuator 
(such as a robot arm) in the control loop impacts the BMI and 
the subject’s performance. Two previous studies have 
reported that macaque monkeys learn to operate a closed- 
loop BMI (BMIc) using visual feedback (Serruya et al. 2002; 
Taylor et al. 2002), but the animals in these studies did not 
control a real mechanical actuator. 

Finally, more data are needed to evaluate the extent, 
relevance, and behavioral meaning of cortical reorganization 
that can be triggered by operation of a BMIc. A possibility of 
such reorganization is supported by results in plasticity in M1 
neurons in a force-field adaptation task (Li et al. 2001) and by 
an initial report of changes in directional selectivity in a small 
sample of M1 neurons during BMIc operation (Taylor et al. 
2002). 

In this paper, we present the results from a series of long- 
term studies in macaque monkeys to address several of the 
fundamental issues that currently shape the debate on BMIs. 
In particular, we demonstrate for what we believe is the first 
time the ability of the same ensemble of cells in closed-loop 
mode to control two distinct movements of a robotic arm: 
reaching and grasping. In addition, we demonstrate how the 
monkeys learn to control a real robotic actuator using a 
BMIc. We also report on how they overcome the robot 
dynamics and return to the same level of performance 
without modification of the task. Finally, we quantitatively 
compare the contribution of neural populations in multiple 
cortical areas needed to create this control and analyze 
changes in these contributions during learning. 


Results 


Using the experimental apparatus illustrated in Figure 1A, 
monkeys were trained in three different tasks: a reaching task 
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(task 1; Figure 1B), a hand-gripping task (task 2; Figure 1B), 
and a reach-and-grasp task (task 3; Figure 1B). 

We used multiple linear models, similar to those described 
in our previous studies (Wessberg et al. 2000), to simulta- 
neously extract a variety of motor parameters (i.e., hand 
position [HPx, HPy, HPz], velocity [HVx, HVy, HVz], and 
gripping force [GF]) and multiple muscle electromyograms 
(EMGs) from the activity of cortical neural ensembles. 
Although all these parameters were extracted in real time 
on each session, only some of them were used to control the 
BMIc, depending on each of the three tasks the monkeys had 
to solve in a given day. In each recording session, an initial 30- 
min period was used for training of these models. During this 
period, monkeys used a hand-held pole either to move a 
cursor on the screen or to change the cursor size by 
application of gripping force to the pole. This period is 
referred to as “pole control” mode. As the models converged 
to an optimal performance, their coefficients were fixed and 
the control of the cursor position (task 1 and 3) and/or size 
(task 2 and 3) was obtained directly from the output of the 
linear models. This period is referred to as “brain control” 
mode. During brain control mode, animals initially produced 
arm movements, but they soon realized that these were not 
necessary and ceased to produce them for periods of time. To 
systematically study this phenomenon, we removed the pole 
after the monkey ceased to produce arm movements in a 
session. In each task, after initial training, a 6 DOF (degree-of- 
freedom) robot arm equipped with a 1 DOF gripper was 
included in the BMIc control loop. In all experiments, visual 
feedback (i.e., cursor position/size) informed the animal about 
the BMIc’s performance. When the robot was used, cursor 
position indicated to the animal the X and Y coordinates of 
the robot hand. The cursor size provided feedback of the 
force measured by the sensors on the robot’s gripper. The 
time delay between the output of the linear model and the 
response of the robot was in the range of 60-90 ms. 

For each task, training continued until the animal reached 
high levels of performance in brain control mode. During this 
learning period, both the animal’s and the BMIc’s perfor- 
mance were assessed using several measures. Chance perfor- 
mance was assessed using Monte Carlo simulations of random 
walks. Contributions of individual neurons and the overall 
contribution of different cortical areas to the prediction of 
multiple motor signals were evaluated. In addition, changes 
in directional tuning of the neurons that resulted from using 
the BMIc were quantified. 


Behavioral Performance during Long-Term 
Operation of a BMIc 

Figure 1C-1E illustrates procedural motor learning as 
animals interacted with the BMIc in each of the three tasks. 
Improvement in behavioral performance with the BMIc was 
indicated by a significant increase in the percentage of trials 
completed successfully (Figure 1C-1E, top graphs) and by a 
reduction in movement time (Figure 1C-1E, bottom graphs). 
For task 1, both monkeys had some training in pole control of 
task 1 (data not shown) several weeks before the series of 
successive daily sessions illustrated in Figure 1C. For both 
tasks 1 and 2, after a relatively small number of daily training 
sessions, the monkeys’ performance in brain control reached 
levels similar to those during pole control (Figure 1C and 1D). 
For tasks 2 and 3, all behavioral data are plotted, given that in 
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Figure 1. Experimental Setup, Behavioral Tasks, Changes in Performance with Training, EMG Records during Pole and Brain Control, and Stability of 
Model Predictions 


(A) Behavioral setup and control loops, consisting of the data acquisition system, the computer running multiple linear models in real time, the 
robot arm equipped with a gripper, and the visual display. The pole was equipped with a gripping force transducer. Robot position was 
translated into cursor position on the screen, and feedback of the gripping force was provided by changing the cursor size. 

(B) Schematics of three behavioral tasks. In task 1, the monkey’s goal was to move the cursor to a visual target (green) that appeared at random 
locations on the screen. In task 2, the pole was stationary, and the monkey had to grasp a virtual object by developing a particular gripping force 
instructed by two red circles displayed on the screen. Task 3 was a combination of tasks 1 and 2. The monkey had to move the cursor to the target 
and then develop a gripping force necessary to grasp a virtual object. 

(C-E) Behavioral performance for two monkeys in tasks 1-3. The percentage of correctly completed trials increased, while the time to conclude a 
trial decreased with training. This was true for both pole (blue) and brain (red) control. Horizontal (green) lines indicate chance performance 
obtained from the random walk model. The introduction of the robot arm into the BMIc control loop resulted in a drop in behavioral 
performance. In approximately seven training sessions, the animal’s behavioral performance gradually returned to the initial values. This effect 
took place during both pole and brain control. 

(F) Stability of model predictions of hand velocity during long pole-control sessions (more than 50 min) for two monkeys performing task 1. The 
first 10 min of performance were used to train the model, and then its coefficients were frozen. Model predictions remained highly accurate for 
tens of minutes. 

(G) Surface EMGs of arm muscles recorded in task | for pole control (left) and brain control without arm movements (right). Top plots show the 
X-coordinate of the cursor; plots below display EMGs of wrist flexors, wrist extensors, and biceps. EMG modulations were absent in brain 


control. 
DOI: 10.1371 /journal.pbio.0000042.¢001 


both cases pole and brain controls were used since the first 
day of training. Behavioral improvement was also observed in 
task 3, which combined elements of tasks 1 and 2 (Figure 1F). 
In all three tasks, the levels of performance attained during 
brain control mode by far exceeded those predicted by a 
random walk model (dashed and dotted lines in Figure 1C- 
1E). Moreover, both animals could operate the BMIc without 
any overt arm movement and muscle activity, as demon- 
strated by the lack of EMG activity in several arm muscles 
(Figure 1G). The ratios of the standard deviation of the 
muscle activity during pole versus brain control for these 
muscles were 14.67 (wrist flexors), 9.87 (wrist extensors), and 
2.77 (biceps). 

A key novel feature of this study was the introduction of 
the robot equipped with a gripper into the control loop of 
the BMIc after the animals had learned the task. Figure 1C 
shows that because the intrinsic dynamics of the robot 
produced a lag between the pole movement and the cursor 
movement, the monkeys’ performance initially declined. With 
time, however, the performance rapidly returned to the same 
levels as seen in previous training sessions (Figure 1C). It is 
critical to note that the high accuracy in the control of the 
robot was achieved by using velocity control in the BMIc, 
which produced smooth predicted trajectories, and by the 
fine tuning of robot controller parameters. These parameters 
were fixed across sessions in both monkeys. The controller 
sent velocity commands to the robot every 60-90 ms. Each of 
these commands compensated for potential position errors 
of the robot hand that resulted from previous commands. 

In all experiments, the animals continuously received visual 
feedback of their performance. Unlike previous results in owl 
monkeys from experiments in which an open-loop BMI was 
implemented (Wessberg and Nicolelis 2003), after the model 
parameters were fixed, its predictions did not drift substan- 
tially from initial best performance, even during 1-h record- 
ings. As shown in the examples of Figure 1F, prediction of 
grasping force (mean + SEM, R=0.84 + 5 X10°-°) in monkey 
1, as well as hand position (R = 0.63 = 3 X 10°) and velocity 
(R = 0.73 + 5 X 10°) in monkey 2, remained very stable 
despite some transient fluctuations (slopes for black, magen- 
ta, and cyan lines are, respectively, —2.16 X 104, —5.15 X 10+, 
and —1.1 X 10~°). One possibility is that the presence of 
continuous visual feedback helped to stabilize model per- 
formance. 
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Which Motor Parameters Can Be Extracted in Real Time? 

Throughout learning of all three behavioral tasks, pop- 
ulations of neurons distributed in multiple frontal and 
parietal cortical areas exhibited task-related modulations of 
their firing rates. Using multiple linear models running in 
parallel, several motor signals were extracted from those 
modulations. To evaluate the performance of the models in 
extracting different motor parameters, the models were first 
trained using 15 min of pole control data and then 
subsequent data were predicted. Figure 2A shows represen- 
tative 1-min records of such predictions of hand position 
(HPx, HPy), hand velocity (HVx, HVy), and gripping force 
(GF). Figure 2B shows the model prediction of EMG activity. 
In well-trained animals, the linear models accounted for up to 
85% of the variance of hand position, 80% of hand velocity, 
95% of gripping force, and 61% of multiple EMG activity. 
These results show that elaborate hand movements, such as 
the ones required to solve task 3, could be predicted from 
brain activity using a BMIc with the simultaneous application 
of multiple linear models. 


What Cortical Areas to Sample? How Many Neurons to 
Record From? What Type of Neuronal Signal to Use? 

Several analytical tools were used to address these 
fundamental questions. By measuring the correlation be- 
tween neuronal firing and each of the predicted parameters 
(Figure 2A), we observed that single neurons located in 
frontal and parietal areas contributed to real-time predic- 
tions of all motor parameters analyzed (Figure 2C). Although 
cortical areas are known to have functional specializations 
(Wise et al. 1997; Burnod et al. 1999), our sample of M1, dorsal 
premotor cortex (PMd), supplementary motor area (SMA), 
posterior parietal cortex (PP), and primary somatosensory 
cortex (S1) cells provided information, albeit at different 
levels, for predictions of hand position, velocity, gripping 
force, and multiple EMGs. 

For each motor parameter analyzed, increasing the size of 
the neuronal population improved the quality of prediction. 
The effect of sample size on predictions was clearly shown 
using neuron-dropping (ND) analysis (Figure 2D-2F). ND 
plots indicate the number of neurons that are required to 
achieve a particular level of model prediction for each 
cortical area. Although all cortical areas surveyed contained 
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Figure 2. Performance of Linear Models in Predicting Multiple Parameters of Arm Movements, Gripping Force, and EMG from the Activity Frontoparietal 


Neuronal Ensembles Recorded in Pole Control 
(A) Motor parameters (blue) and their prediction using linear models (red). From top to bottom: Hand position (HPx, HPy) and velocity (HVx, 
HVy) during execution of task 1 and gripping force (GF) during execution of tasks 2 and 1. 


(B) EMGs (blue) recorded in task | and their prediction (red). 
(C) Contribution of neurons from the same ensemble to predictions of hand position (top), velocity (middle), and gripping force (bottom). 


Contributions were measured as correlation coefficients (R) between the recorded motor parameters and their values predicted by the linear 
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model. The color bar at the bottom indicates cortical areas where the neurons were located. Each neuron contributed to prediction of multiple 
parameters of movements, and each area contained information about all parameters. 

(D-F) Contribution of different cortical areas to model predictions of hand position, velocity (task 1), and gripping force (task 2). For each area, 
ND curves represent the average prediction accuracy (R°) as a function of number of neurons needed to attain it. Contributions of each cortical 
area vary for different parameters. Typically, more than 30 randomly sampled neurons were required for an acceptable level of prediction. 
(G-I) Comparison of the contribution of single units (blue) and multiunits (red) to predictions of hand position, velocity, and gripping force. 
Single units and multiunits were taken from all cortical areas. Single units’ contribution exceeded that of multiunits by approximately 20%. 


DOI: 10.137 1/journal.pbio.0000042.g002 


information about any given motor parameter, for each area, 
different numbers of neurons were required to achieve the 
same level of prediction. For example, the sample of M1 
neurons (33-56 cells) taken alone (Figure 2D-2F) was the best 
predictor for all motor variables (73% of the variance for 
hand position, 66% for velocity, 83% for gripping force). The 
sample of SMA neurons (16-19 cells) produced high 
predictions of hand position (51%) and velocity (51%), but 
poor prediction of gripping force (19%). The activity of PMd 
(64 cells) or S1 (21-39 cells) ensembles predicted hand 
position (48% for both PMd and S1) and velocity (46% for 
PMd and 35% for S1) reasonably well, but yielded lower 
predictions of GF (29% for PMd neurons and 15% for S1). 
Meanwhile, the sample from PP (63-64 cells) yielded very 
accurate predictions of gripping force (73%) and hand 
velocity (52%), but not hand position (25%). Ensemble 
predictions of gripping force in most cases were more 
accurate than those obtained from the same population for 
hand position and velocity. In addition, the ND analysis 
revealed that predictions of any motor parameter based on 
combined neural ensemble activity were far superior to those 
obtained based only on the mean contribution of single 
neurons. 

Another interesting finding emerged from the comparison 
of the contribution of single-unit versus multiunit activity to 
the performance of the linear models. Overall, up to 90 single 
units and 95 multiunits were simultaneously recorded in 
monkey | and 75 single units and 175 multiunits in monkey 2. 
The cell population was stable not only during the length of 
the recording sessions but across sessions. The vast majority 
of the population remained stable for several weeks and, in 
some cases, months (Nicolelis et al. 2003). 

Figure 2G-2I shows that the linear predictions of hand 
position, velocity, and grasping force were somewhat better 
when single units were used (by 17%, 20%, and 17%, 
respectively). That difference could be compensated by 
increasing the number of channels. For example, as seen in 
Figure 2G, around 30 additional multiunits compensate for 
the difference in prediction of hand position provided by 20 
single units. That difference was, however, not critical, as the 
animals could still maintain high levels of BMI performance 
in all tasks using multiunit activity only. Thus, in contrast to 
previous studies (Serruya et al. 2002; Taylor et al. 2002) that 
dealt with fewer motor parameters and a simpler task, we 
observed that large neuronal samples were needed to achieve 
a high level of real-time prediction of one or more motor 
parameters and, consequently, high behavioral proficiency in 
operating the BMIc. 


Functional Cortical Reorganization during 
Operation of BMIc 

The achievement of high proficiency in the operation of 
the BMIc by the monkeys was consistent with procedural 
motor learning. Since cortical ensemble recordings were 
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obtained during behavioral training in all three tasks, it was 
possible to examine the neurophysiological correlates of this 
learning process. Both short (within a recording session) and 
long-term (across recording sessions) physiological modifica- 
tions took place. 

Long-term functional changes in multiple cortical areas 
were evident in both animals. For instance, the average 
contribution of single neurons to model performance 
increased with learning. Figure 3A shows changes in the 
contribution of single cortical neurons (measured in terms of 
correlation coefficient, R, color-coded, where blue shows low 
R; red, high R) from five cortical areas (PMd, M1, S1, SMA, 
and M1 ipsilateral) to the linear model that predicted hand 
position in task 1. Data from 42 recording sessions are shown. 
In these sessions, predictions of hand position (HPx, HPy) 
were used to control the cursor on the screen. By the end of 
the training, very accurate predictions of hand position and 
velocity were obtained (mean R + SEM; HPx = 0.75 + 0.04, 
HPy = 0.72 + 0.04, HVx = 0.70 + 0.03, and HVy = 0.71 + 
0.02). These high values were reached through a significant 
increase in contribution of individual neurons to the linear 
model. When the mean contribution of single neurons was 
plotted as a function of their cortical area location, differ- 
ences across cortical areas were found (Figure 3B-3E). The 
change was higher in SMA (Figure 3E; R = 0.81, slope = 0.01, p 
< 0.001) than in PMd (Figure 3B; R = 0.81, slope = 1 X 10°, p 
< 0.001), S1 (Figure 3D; R = 0.67, slope = 4 X 10°°, p < 0.001), 
and M1 (Figure 3C; R= 0.50, slope = 3 X 10°, p < 9.001). Note 
that from the beginning of training, M1 neurons (Figure 3C) 
provided the highest mean contribution. By the end of 42 
sessions, however, the mean contribution of neurons located 
in other cortical areas (e.g., SMA, PMd, and S1) was as high as 
that of M1. It is noteworthy that the significant enhancement 
in contribution occurred for the model predicting hand 
position (average of all cortical areas, R = 0.80, slope = 4 X 
10%, p < 9.001), but not the one predicting hand velocity (R= 
0.05, slope = 2.2 X 10+). This selectivity coincided with the 
use of a position model in the BMIc during these 42 sessions. 
Thus, long-term training with the BMIc using a particular 
model could result in selective enhancement of the mean 
contribution of neurons to that model, but not the others. 


Changes in Neuronal Direction Tuning during 
Operation of a BMIc 

As animals learned to operate the BMIc, we also observed 
short-term changes in neuronal directional tuning, within a 
single recording session, after switching the BMIc mode of 
operation from pole to brain control. Directional tuning 
curves (DTCs) reflected dependency of the neuronal firing 
rate on movement direction of either the pole or the cursor. 
DTCs were normalized by dividing average firing rates for 
particular movement directions by the standard deviation of 
the whole firing rate record and then subtracting the DTC 
mean. Using that normalization, changes in firing rate with 
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(A) Color-coded (red shows high values; blue, low values) representation of individual contributions measured as the correlation coefficient (R) 
of neurons to linear model predictions of hand position for 42 training sessions. The average contribution steadily increased with training. The 
bar on the left indicates cortical location of the neurons. 

(B-E) Average contribution of neurons located in different cortical areas (PMd, M1, S1, and SMA, respectively) to hand position prediction 
during 42 recording sessions. 

(F) Average contribution for the whole ensemble to hand position prediction versus hand velocity predictions. A linear increase in contribution 


was observed only for predictions of hand position. 
DOI: 10.137 1/journal.pbio.0000042.g003 


movement direction were compared with the overall varia- 
tion of firing rate. Average directional tuning of neural 
ensembles (DTE) was also assessed, and the spread of the 
preferred tuning directions was evaluated as the ensemble 
average of the angles between preferred directions in pairs of 
neurons. Color-coded population histograms (Figure 4A-4D) 
displayed the DTCs of all recorded neurons. Polar plots 
(magenta figures in Figure 4A-4D) showed the DTE and 
preferred direction spread. Figure 4A-4D shows that DTCs 
and DTEs for the same neural ensemble had distinct features 
during pole control (Figure 4A), during brain control with the 
presence of arm movements (Figure 4B and 4D), and during 
brain control without arm movements (Figure 4C). Even if the 
animal was still making arm movements after switching to 
brain control and direction tuning was calculated in relation 
to pole movements (compare Figure 4A with 4D), DTC and 


PLoS Biology | http://biology.plosjournals.org 


DTE changed significantly when compared to curves obtained 
during pole control (R = 0.57 using pole movements as a 
reference direction, R = 0.70 using cursor movements as a 
reference). The changes in DTC and DTE became greater as 
the animal ceased to produce arm movements in brain 
control (Figure 4C) (R = 0.48). Notice, however, that the 
pattern for brain control without arm movements (Figure 4C) 
was also distinct from that for brain control with arm 
movements (Figure 4B) (R = 0.57). These findings suggest that 
both the cursor and pole movements influenced the 
definition of directional tuning profiles in multiple cortical 
areas. 

After the mode of operation was switched to brain control, 
pole and cursor movements became dissociated. Further, as 
animals started controlling the BMIc without producing overt 
hand movements, directional tuning primarily reflected 
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Figure 4. Directional Tuning in Frontoparietal Ensemble during Different Modes of Operation in Task 1 


(A-D) Directional tuning during pole control (A), brain control with arm movements (tuning assessed from cursor movements) (B), brain control 
without arm movements (tuning assessed from cursor movements) (C), and brain control with arm movements (tuning assessed from pole 
movements) (D). In each of the color-coded diagrams (red shows high values and blue low values; see color scale), the rows depict normalized 
directional tuning for individual cells. Because of the high directional tuning values of some cells (e.g., that shown in [H]), a color scale limit was 
set at 0.3 to allow color representation of the largest possible number of cells. Each tuning curve contains eight points that have been 
interpolated for visual clarity. Correspondence of tuning patterns under different conditions has been quantified using correlation coefficients 
(shown near lines connecting the diagrams). The highest correspondence was between tuning during pole control and brain control with arm 
movements. A much less similar pattern of direction tuning emerged during brain control without arm movements. Polar plots near each 
diagram show average directional tuning for the whole neural ensemble recorded. They indicate an average decrease in tuning strength and 
shifts in the preferred direction of tuning as the operation mode was switched from pole to brain control. Spread of preferred directions (90° 
corresponds to uniformly random distribution) is indicated near each polar plot. 

(E-G) Scatterplots comparing DTD (maximum minus minimum values of tuning curves) during pole control versus brain control with and 
without arm movements. DTD during brain control was consistently lower than during pole control. This effect was particularly evident during 
brain control without arm movements. 

(H-J) Changes in directional tuning for individual neurons under different conditions. Blue shows pole control; red, brain control with arm 
movements (tuning assessed from pole movements); and green, brain control without arm movements. The first illustrated cell (H) was tuned 
only when the monkey moved its arm, more so during pole control. The second cell (I) had similar tuning during all modes of operation, but 
tuning depth was the highest for pole control and the lowest for brain control without arm movements. The third cell (J) was better tuned during 


brain control. 
DOI: 10.1371 /journal.pbio.0000042.g004 


cursor movements. Interestingly, during the transition from 
pole to brain control, directional tuning depth (DTD) was 
reduced for most cells. Figure 4E-4G shows this effect by 
comparing the DTD in individual neurons during pole 
control (Y axis of Figure 4E-4G) and brain control (X axis). 
Notice that the reduction in tuning depth was more 
pronounced when no arm movements were produced during 
brain control (Figure 4G). Reduction in directional tuning 
during brain control with no movements characterized 68% 
of the sampled neurons and included neurons from all 
cortical areas (see color dots in Figure 4E-4G and examples of 
Figure 4H and 41). A small percentage of neurons (14%) did 
not show such change. Perhaps more surprisingly, a fraction 
of neurons (18%) enhanced their directional tuning during 
the switch from pole to brain control (see Figure 4J). These 
neurons correspond to the dots that are located to the right 
of the main diagonal in Figure 4E-4G. 

Operating the BMIc without making movements was 
characterized by an appearance of peculiar patterns of 
directional tuning at the population level. Figure 5A and 
5C displays the evolution of DTC and DTE for the same 
neural ensemble during four task 1 sessions with the robot in 
the loop. Whereas in each case DTCs during brain control 
resembled those in pole control, they evolved toward a more 
organized distribution. Although certain diversity in DTCs 
remained, clear groups of neurons sharing similar DTCs 
appeared as a result of training (Figure 5C). Quantitatively, 
this effect was manifested by a decrease in the spread of 
preferred directions. This effect was also evident in the polar 
plots showing population-average tuning (i.e., DTE). The DTE 
became progressively sharper and rotated clockwise. 
Throughout the four sessions depicted in Figure 5, tuning 
depth remained higher during pole than brain control 
operation of the BMIc (Figure 5B). 

Further analysis revealed that significant changes in direc- 
tional tuning also occurred within a single recording session 
during brain control (Figure 5D). The session illustrated in 
Figure 5D was characterized by a gradual improvement in 
behavioral performance during brain control without arm 
movements, as evident from measurements made every 5 min 
(Figure 5E). The population histograms of Figure 5D show 
that the distribution of DTCs, although variable, became on 
average tighter across all cortical areas, defining a vertical 
band across the population histogram. This tightening was 
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manifested by a decrease in the spread of preferred 
directions (Figure 5F). Moreover, average tuning depth 
gradually increased (Figure 5G), but remained lower than 
that observed during pole control. 

Similarity of DTCs during brain control indicated that 
particular movement directions were associated with simul- 
taneous increases in activity in many neurons; i.e., firing rates 
of these neurons became more correlated. Indeed, we found 
increases in broad correlation (100 ms time window) in 
neuronal firing within and between cortical areas. Thus, 
during the transition from pole to brain control in task 1, the 
average correlated firing between pairs of cortical neurons, 
measured in terms of correlation coefficient (mean + SEM), 
increased from 0.02 + 1 X 10°-* to 0.06 + 2X 10“, a 3-fold 
rise that was highly significant (Wilcoxon signed rank test, p < 
0.0001). All cortical areas tested (M1, PMD, SMA, S1, and M1 
ipsilateral) showed increases in correlated firing. The highest 
within-area increases from pole to brain control were 
observed in M1 (Agrain-Pote = 0.07), S1 (Agrain-Pole = 0.05), 
and PMd (Agyain-Pole = 0.03). The highest between-area 
increases were observed between M1-S1 (Agrain-Pole = 0.06, 
M1-PMd (Agrain-Pole = 0.04), PMd-S1 (Agrain-Pote = 0.04), M1- 
SMA (Agrain-Pole = 0.02), and M1 contra~M Lipsi (Agrain-Pole = 
0.02). 

Changes in average firing rates of the neurons during 
switching from pole to brain control were insubstantial. 
Firing rates of individual cells ranged from 0.1 to 40 spikes/s 
(8 + 8 spikes/s; mean + SD). After the mode was switched to 
brain control and the monkey continued to move the arm, 
firing rates increased on average by 4% from pole-control 
level. When the monkey controlled the BMI without moving 
the arm, the average neuronal firing rates decreased 2.5% 
from pole-control level. 


Real-Time Prediction of Gripping Force 

In addition to reproducing hand trajectories with great 
accuracy, linear models also allowed the reconstruction of 
fine variations in gripping force produced by both monkeys 
in tasks 2 and 3. Figure 6A shows that during execution of 
task 2, most of the recorded cortical neurons contained 
information about gripping force. In this figure, normal- 
ization was achieved by dividing the firing rate of each 
individual neuron by its standard deviation. In this way, 
force-related modulations are expressed relative to the 
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Figure 5. Plasticity of Directional Tuning during Training in Brain Control without Arm Movements 


Conventions are as in Figure 4. 

(A) Directional tuning profiles during four sessions in pole control (task 1). Percentages of correctly performed trials are indicated for each 
session. 

(B) Scatterplots comparing directional tuning during pole versus brain control for the same sessions. For each day, DTD was on average higher in 
pole control. 

(C) Directional tuning during brain control for the same sessions as in (A). Note the emergence of a population pattern in which a group of 
neurons (with some exceptions) exhibits a similar preferred direction. This is manifested by a decrease in the spread of preferred directions 
(shown near polar plots). Notice also a gradual rotation of the population preferred direction (see polar plots) with training. 

(D) Gradual changes in DTE during one representative session of brain control without arm movements. This 60-min session was split into 5-min 


periods, five of which are shown. 


(E) Improvement in behavioral performance during a single session (same as in [D]) . 


(F) Decrease in the spread of preferred directions during that session. 


(G) Increase in average tuning depth during the same session. 
DOI: 10.137 1/journal.pbio.0000042.¢005 


overall variability of the neuron’s firing rate. Both monkeys 
mastered task 2 in seven to eight sessions. Figure 6B displays 
the evolution of the average contribution of neurons from 
different areas of monkey 1 to model predictions during this 
period. Contribution of contralateral M1 (R = 0.77, slope = 
0.02, p < 0.05) and S1 (R = 0.85, slope = 0.02, p < 0.002) 
increased significantly, while that of PMd (R = 0.19, slope = 2 
xX 10%), SMA (R = 0.34, slope = 0.01), and ipsilateral M1 (R = 
0.38, slope = —0.01) did not change substantially. For the 
whole ensemble combined, there was a significant increase in 
contribution in both monkey 1 (R = 0.95, slope = 0.02, p < 
0.001) and monkey 2 (R = 0.54, slope = 0.01, p < 0.05). By 
comparing Figure 3B-3F and Figure 6B, we can see that while 
M1 and S1 neurons showed changes during both tasks 1 and 2, 
PMd and SMA neurons showed changes in task 1, but not in 
task 2. This may reflect the greater involvement of these 
cortical areas in learning visuomotor spatial relationships 
than in the production of muscle force. 

Switching from pole to brain control did not affect 
neuronal firing rate correlations in task 2. This could be 
related to saturation of this parameter because the average 
firing rate correlation observed during pole control when 
monkeys performed task 2 (0.056) was already higher than 
that observed during task 1 (0.022, p < 0.001, Wilcoxon rank 
sum test). 


Using a BMIc to Reach and Grasp Virtual Objects 

Our experiments demonstrated, to our knowledge for the 
first time, that monkeys can learn to control a BMIc to 
produce a combination of reaching and grasping movements 
to locate and grasp virtual objects. The major challenge in 
task 3 was to simultaneously predict hand position and 
gripping force using the activity recorded from the same 
neuronal ensemble. This problem could not be reduced to 
predicting only hand position as in task 1 or gripping force in 
task 2, because the animal had to sequentially reach and grasp 
the target. 

The DTD of cortical neurons, measured during pole 
control, increased almost linearly during the learning of task 
3 (Figure 6C). Although this effect was significant in all 
cortical areas tested, its magnitude varied across areas. The 
most prominent increase in DTD was observed in M1 (red 
dots in Figure 6C; R = 0.81, slope = 0.02). Neurons in S1 
(green dots in Figure 6C; R = 0.82, slope = 0.02) and ipsilateral 
M1 (magenta dots in Figure 6C; R = 0.81, slope = 0.02) 
exhibited the next largest increase. Relatively smaller DTD 
increases were observed in PP (cyan dots in Figure 6C; R = 
0.73, slope = 0.01), SMA (black dots in Figure 6C; R = 0.63, 
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slope = 0.01), and PMd (blue dots in Figure 6C; R= 0.51, slope 
= 0.01). Similar to task 1, tuning depth was higher during pole 
control than during brain control. As in task 1, the DTC and 
DTE patterns changed during training (data not shown). 
Improvements in model performance occurred as well. 
Figure 6D and 6E show the evolution in accuracy of real- 
time predictions of hand position, velocity, and gripping 
force during 14 sessions for both monkeys. During this 
period, real-time predictions for both hand position and 
velocity improved with training, while predictions of gripping 
force remained high and stable in two monkeys (monkey 1, 
mean + SD, R= 0.86 + 0.04; monkey 2, R= 0.79 + 0.03). 

The monkeys’ performance in brain control in task 3 
approximated that during pole control, with characteristic 
robot displacement (reach) followed by force increase (grasp). 
Figure 6F and FG shows several representative examples of 
reaching and grasping during pole and brain control in task 3 
by monkey 1. Hand position (X, Y) and gripping force (F) 
records are shown. In the display of hand trajectories, the size 
of the disc at the end of each hand movement shows the 
gripping force produced by the monkey (Figure 6F) or by the 
BMIc (Figure 6G) to grasp a virtual object. The reach (r) and 
grasp (g) phases are clearly separated, demonstrating that the 
monkeys could use the same sample of neurons to produce 
distinct motor outputs at different moments in time. Thus, 
during the reaching phase, X and Y changed, while F 
remained relatively stable. However, as the monkey got closer 
to the virtual object, F started to increase, while X and Y 
stabilized to maintain the cursor over the virtual object. Thus, 
our goal to train the monkey to reproduce coupled sequences 
of reach-and-grasp movements using the BMIc was accom- 
plished. 


Discussion 


Reliable, long-term operation of a BMIc was achieved by 
extracting multiple motor parameters (i.e., hand position, 
hand velocity, and gripping force) from the simultaneously 
recorded activity of frontopariental neural ensembles. Ma- 
caque monkeys learned to use the BMIc to reach and grasp 
virtual objects with a robot even in the absence of overt arm 
movements. Accurate performance was possible because 
large populations of neurons from multiple cortical areas 
were sampled. Thus, the present study shows that large 
ensembles are preferable for efficient operation of a BMI. 
This conclusion is consistent with the notion that motor 
programming and execution are represented in a highly 
distributed fashion across frontal and parietal areas and that 
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Figure 6. Ensemble Encoding of Gripping Force, Plasticity of Directional Tuning, and Neuronal Contribution to Model Performance during Learning to 


Control the BMIc for Reaching and Grasping 

(A) Perievent time histograms (PETHs) in task 2 for the neuronal population sampled in monkey 1. The plots on top are color-coded (red shows 
high values; blue, low values). Each horizontal row represents a PETH for a single-neuron or multiunit activity. PETHs have been normalized by 
subtracting the mean and then dividing by the standard deviation. PETHs are aligned on the gripping force onset (crossing a threshold). Plots at 
the bottom show the corresponding average traces of gripping force. Note the general similarity of PETHs in pole (left) and brain (right) control 
in this relatively easy task. Cortical location of neurons is indicated by the bar on the top left. Note the distinct pattern of activation for different 


areas. 
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(B) Changes in the mean contribution of neurons from different cortical areas to model predictions during training of monkey | in task 2. 
(C) Increases in directional tuning for six cortical areas during training in pole control in task 3. 
(D and E) Increases in neuronal contribution to linear models predicting hand position (blue), hand velocity (red), and gripping force (black) 


during learning task 3 in both monkeys. 


(F and G) Representative robot trajectories and gripping force profiles in an advanced stage of training in task 3 during both pole and brain 
control. The bottom graphs show trajectories and the amount of the gripping force developed during grasping each virtual object. The dotted 
vertical lines in the panels indicate the end of reach phase and the beginning of grasp phase. Note that during both modes of BMIc operation, 
the patterns of reaching and grasping movements (displacement followed by force increase) were preserved. 


DOI: 10.1371 journal.pbio.0000042.g006 


each of these areas contains neurons that represent multiple 
motor parameters. We suggest that, in principle, any of these 
areas could be used to operate a BMI, provided that a large 
enough neuronal sample was obtained. While analysis of ND 
curves (see Figure 2D-2F) indicates that a significant sample 
of M1 neurons consistently provides the best predictions of 
all motor parameters analyzed, neurons in areas such as SMA, 
S1, PMd, and PP contribute to BMI performance as well. 

Our argument for using large neuronal samples is also 
supported by the fact that some neurons can be lost during 
chronic recordings, either due to electrode malfunction, 
modification of electrode tip properties, or simple cell death. 
Then, a BMI that relies on only very small samples of neurons 
(e.g., 8-30 cells) (Serruya et al. 2002; Taylor et al. 2002), all 
derived from a single cortical area, would not be able to 
provide a broad variety of motor outputs, handle changes in 
cortical properties, or cope with alterations in the neuronal 
sample over time. 

Another important finding of this study is that accurate 
real-time prediction of all motor parameters as well as a high 
level of BMI control can be obtained from multiunit signals. 
This observation is essential because it eliminates the need to 
develop elaborate real-time spike-sorting algorithms, a major 
technological challenge, for the design of a future cortical 
neuroprosthesis for clinical applications. 

Our experiments also demonstrate that the initial intro- 
duction of a mechanical device, such as the robot arm, in the 
control loop of a BMIc significantly impacts learning and task 
performance. After the robot was introduced in the control 
loop, the monkey had to adjust to the dynamics of this 
artificial actuator. As a result, there was an immediate drop in 
performance (see Figure 3G). With further training, however, 
the animals were able to overcome the difficulties. Thus, the 
simple utilization of the output of a real-time model to move 
a cursor on a computer screen (Serruya et al. 2002; Taylor et 
al. 2002) does not fully test the limitations and challenges 
involved in operating a clinically relevant BMI. Instead, such 
testing must include the incorporation in the apparatus of 
the mechanical actuator designed to enact the subject’s motor 
intentions and training the subject to operate it. 

As was proposed recently (Nicolelis and Ribeiro 2002), 
multisite, multielectrode recordings (Nicolelis et al. 2003) also 
allowed us to quantify neurophysiological modifications 
occurring in cortical networks, as animals learned motor 
tasks of different complexity. At a single-neuron level, one 
modification observed was a reduction in the strength of 
directional tuning as animals switched from pole to brain 
control of the BMI, an effect that reached its maximum as 
animals ceased to produce overt arm movements. This 
finding touches directly on the ongoing debate of two 
opposing views of what the motor cortex encodes (Mussa- 
Ivaldi 1988; Georgopoulos et al. 1989; Kakei et al. 1999; 
Todorov 2000; Johnson et al. 2001; Scott et el. 2001). At the 
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first glance, the reduction in tuning depth in the absence of 
arm movements could be interpreted as providing support to 
the notion that directional tuning in the motor cortex is 
highly influenced by movement dynamics. Thus, the elimi- 
nation of proprioceptive feedback during brain control could 
explain the significant reduction in directional tuning. 
However, a smaller but significant decrease in directional 
tuning was also observed during brain control while animals 
still used their hands to move the pole. This suggests that 
directional tuning reflects neither movement dynamics nor 
abstract motor goals alone, but rather their combination. 
Additional findings support this contention. For example, a 
small fraction of M1 and S1 neurons became better direc- 
tionally tuned when the monkey did not make hand move- 
ments during brain control (see Figure 4Ba—4Bd and Figure 
5G). Moreover, during brain control there was a significant 
increase in pairwise-correlated firing and a tendency for 
groups of neurons to exhibit rather similar DTCs (see Figures 
4 and 5). Increases in tuning depth accompanied improve- 
ments in performance during brain control, although values 
observed during pole control were never reached (see Figure 
5E and 5G). All together, these physiological changes suggest 
that as animals learn to operate the BMI during brain control, 
visual feedback signals representing the goal of movement, 
rather than information about arm movements per se, 
become the main guiding signal to the cortical neurons that 
drive the BMIc. Thus, we hypothesize that, as monkeys learn 
to formulate a much more abstract strategy to achieve the 
goal of moving the cursor to a target, without moving their 
own arms, the dynamics of the robot arm (reflected by the 
cursor movements) become incorporated into multiple 
cortical representations. In other words, we propose that 
the gradual increase in behavioral performance during brain 
control of the BMI emerged as a consequence of a plastic 
reorganization whose main outcome was the assimilation of 
the dynamics of an artificial actuator into the physiological 
properties of frontoparietal neurons. This hypothesis is 
consistent with previous observations in paralyzed humans 
who learned to move a cursor on the screen using cortical 
activity (Kennedy and King 2000). It is also supported by the 
results that cortical neurons may modulate their firing rate 
either during use of tools (Iriki et al. 1996), according to 
cursor movement on the screen rather than underlying arm 
movements (Alexander and Crutcher 1990; Shen and 
Alexander 1997), or in relation to the orientation of spatial 
attention (Lebedev and Wise 2001). 

Our results on cortical reorganization are very distinct 
from a previous claim of plastic changes in directional tuning 
of cortical cells during the use of BMI (Taylor et al. 2002). 
First, in that previous report, the population vector model 
yielded poor predictions when applied to activity of a small 
sample (n = 18) of M1 cells. Introduction of visual feedback 
improved the subject’s performance to a point in which 
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monkeys could use a BMI to produce stereotypic center-out 
movements of a cursor. The authors claimed that changes in 
cell-preferred direction occurred after switching to brain 
control. However, preferred directions were derived not from 
the real-movement directions of the hand or the cursor, but 
rather from ideal directions defined by target locations. In 
addition, a wide 420-990 ms time window was used to 
measure firing rates. This window was comparable to move- 
ment duration. Therefore, differences in movement trajecto- 
ries and duration between hand and brain control, rather 
than true differences in cell directional tuning, could lead to 
different estimates of preferred direction. The report also 
claims that tuning strength increased with training in brain, 
but not hand, control. However, tuning depth was evaluated 
by measuring covariation between firing rate modulations 
and target locations, rather than actual movement trajecto- 
ries. Because during training, cursor trajectories gradually 
approached a straight line connecting the starting point and 
the target, the observed improvement in covariation between 
target locations and neuronal firing rate modulations could 
simply reflect the improvement in movement accuracy. These 
considerations should be taken into account to decide how 
much of the plasticity reported by Taylor et al. (2002) reflects 
real cortical reorganization instead of resulting from the 
improvement in the animal’s behavioral performance during 
the task used to measure directional tuning. 

In the present study, all the changes in firing and tuning 
properties of neuronal ensembles were related to the actual 
trajectories produced by the monkeys during pole and brain 
control. Moreover, the relationship between the neuronal 
firing and movement patterns was evaluated continuously. 
Thus, the cortical changes reported here more closely 
reflected the relationship between neuronal signals and 
motor behaviors that they underlie. 

Overall, the present findings demonstrate that it is 
reasonable to envision that a cortical neuroprosthesis for 
restoring upper-limb movements could be implemented in 
the future, following the basic BMIc principles described 
here. We propose that long-term operation of such a device 
by paralyzed subjects would lead, through a process of 
cortical plasticity, to the incorporation of artificial actuator 
dynamics into multiple brain representations. Ultimately, we 
predict that this assimilation process will not only ensure 
proficient operation of the neuroprosthesis, but it will also 
confer to subjects the perception that such apparatus has 
become an integral part of their own bodies. 


Materials and Methods 


Behavioral training and electrophysiology. Two adult female 
monkeys (Macaca mulatta) were used in this study. All procedures 
conformed to the National Research Council’s Guide for the Care and 
Use of Laboratory Animals (1996) and were approved by the Duke 
University Animal Care and Use Committee. 

At the time of surgery, animals had completed a period of chair 
training, and one of them (monkey 2) was familiarized with the 
requirements of task | (a large target size was used in this preliminary 
training). Multiple arrays containing 16-64 microwires each (separa- 
tion between adjacent microwires = 300 fm) were implanted in 
several frontal and parietal cortical areas in each animal (Nicolelis et 
al. 2003) (96 in monkey 1 and 320 in monkey 2). Implanted areas 
included the dorsal premotor cortex (PMd), supplementary motor 
area (SMA), and the primary motor cortex (M1) in both hemispheres. 
In monkey | an implant was placed in the primary somatosensory 
cortex (S1). In monkey 2, the medial intraparietal area (MIP) of the 
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posterior parietal cortex (PP) was also implanted. The monkeys 
performed the tasks with their left arms, which were contralateral to 
the areas with the best cell yield. Upon recovery from this surgical 
procedure, animals were transferred to the experimental apparatus 
illustrated in Figure 1A and started behavioral training. 

Monkeys were seated in a primate chair facing a computer 
monitor. They were trained to perform three different tasks using a 
hand-held pole equipped with a pressure transducer (PCB Piezo- 
electrics Inc., Depew, New York, United States) for measuring 
grasping force. The position of the monkey’s hand was obtained 
from an infrared marker located on top of the pole. The marker was 
monitored by an optical tracking system (Optotrak, Northern Digital, 
Waterloo, Ontario, Canada). In the first task, the monkeys were shown 
a small disk (the “cursor” ) and a larger disk (the “target” ). They had 
to use the pole to put the cursor over the target and remain within it 
for 150 ms. Should the monkey cross the target too fast, the target 
disappeared and the trial was not rewarded. Each trial began with a 
target presented in a random position on the screen. The monkeys 
had 5s to hit the target and get a juice reward. In the second task, the 
monkeys were presented with the cursor in the center of the screen 
and two concentric circles. The ring formed by these two circles 
instructed the amount of gripping force the animals had to produce. 
The pole was fixed, and the cursor grew in size as the monkey gripped 
the pole, providing continuous visual feedback of the gripping force. 
Force instruction changed every trial. The third task contained 
elements of tasks 1 and 2. In this task, the monkeys were presented 
with the cursor, the target, and the force-instructing ring and were 
required to manipulate the pole to put the cursor over the target and 
match the ring size by developing the proper amount of gripping 
force, while staying inside the target. The monkeys received juice 
rewards for correct performance. In task 1, the monkeys were initially 
trained without the robot in the loop, but after a certain number of 
sessions, the robot was incorporated to the loop. In tasks 2 and 3, the 
robot was always part of the loop. 

A 512-multichannel acquisition processor (Plexon Inc., Dallas, 
Texas, United States) was employed to simultaneously record from 
single neuron and multiunit activity during each recording session. 
EMG signals were recorded from the skin surface just above the belly 
of the wrist flexors, wrist extensors, and biceps muscles using gold 
disc electrodes (Grass Instrument Co., West Warwick, Rhode Island, 
United States) filled with conductive cream. These signals were 
amplified (gain, 10,000X), high-pass filtered, rectified, and smoothed 
(kernel convolution). 

Linear model. Hand position, velocity, and gripping force were 
modeled as a weighted linear combination of neuronal activity using 
a multidimensional linear regression or Wiener filter, the basic form 
of which is 


y(t) =b+ S7 a(u)x(t — u) + &(t) 


u=—m 


In this equation, x(¢ — u) is an input vector of neuronal firing rates at 
time ¢ and timelag u, y(t) is a vector of kinematic and dynamic 
variables (e.g., position, velocity, gripping force) at time ft, a(u) is a 
vector of weights at timelag u, b is a vector of y-intercepts, and £(t) are 
the residual errors. The lags in the summation can in general be 
negative (in the past) or positive (in the future) with respect to the 
present time ¢. We only considered lags into the past. 
This equation can be recast in matrix form as 


Y=XA, 


here each row in each matrix is a unit of time and each column is a 
data vector. Note that matrix X contains lagged data and thus has a 
column for each lag multiplied by the number of channels; e.g., 100 
channels and 10 lags imply 1000 columns. The y-intercept is handled 
by prepending a column of ones to matrix X. Matrix A is then solved 


by 
A = inv(X’X)X"Y 


Real-time predictions of motor parameters. Predictions of hand 
trajectory and grasping force were generated using the Wiener filter 
described above. Neuronal firing rates were sampled using 100 ms 
bins, and 10 bins preceding a given point in time were used for 
training the model and predicting with it. Models were trained with 
10 min of data and tested by applying them to subsequent records. In 
individual neuron analysis, a model was trained using single-unit/ 
multiunit activity only and then tested for predictions of motor 
parameters. In velocity mode, the model was trained using velocity 
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derived from position measurements by digital differentiation. 
During brain control, predicted velocity was digitally integrated to 
provide an output position signal. To avoid slow drifts, this signal was 
high-pass filtered with a first-order Butterworth filter (cutoff 
frequency of 2 Hz). The linear models independently predicted X, 
Y, and Z hand position coordinates. However, because the three tasks 
reported in this study took place in the X-Y plane, the predictions of 
position and velocity along the Z axis were not used. Several 
alternative decoding algorithms were tested offline, including a 
Kalman filter, normalized least-mean squares filter, and a feed- 
forward backpropagation artificial neural network. None of these 
methods could consistently outperform the Wiener filter. 

Robot arm and gripper. A 6 DOF robotic arm equipped with a 1 
DOF gripper (The ARM, Exact Dynamics, Didam, The Netherlands) 
was used in this study. The gripper was sensorized with pressure 
transducers (Flexiforce, Tekscan, Boston, Massachusetts, United 
States) of 1 lb (2.2 kg) force range for providing grasping force 
feedback. Position feedback of the robot was obtained through the 
built-in joint encoders. Both the commands for controlling the robot 
and the feedback were in Cartesian coordinates. The communication 
between the client computer and the robot was performed via the 
CAN bus (National Instruments, Austin, Texas, United States) 
(sampling period, 60 ms). For the tasks involving grasping, the 
gripper had a light object inserted made of foam material. This object 
was squeezed by the gripper in proportion to either the force applied 
by the monkey in the pole or to the brain signal. 

Data analysis. The monkeys’ behavior was continuously monitored 
and videotaped throughout each recording session. The percentage 
of correctly performed trials and the time to accomplish each trial, 
during both pole and brain controls, were used as measures of 
performance. Chance performance for each task was determined 
using Monte Carlo simulations of a random walk with 2, 1, and 3 DOF 
(for task 1, task 2, and task 3, respectively.) The velocity of the random 
walk was varied from 1 to 500 mm/s, with 10,000 trials for each 
velocity. For tasks | and 3, this was the velocity of the cursor; for task 
2, velocity corresponded to the rate in change of the cursor radius. 
Because each monkey operated the pole at different speed, 
predictions shown in Figure 1C-1E are based on the average velocity 
across all sessions for a given monkey. 

Random neuron-dropping (ND) technique was implemented as 
described by Wessberg et al. (2000). Population data (10 min) were 
used to fit a linear model, which was used to predict motor 
parameters from the subsequent record. A single neuron was then 
randomly removed from the population, the model retrained, and 
new predictions generated. This process was repeated until only one 
neuron remained. The average squared linear correlation coefficient 
(R’) as a function of number of neurons was obtained by repeating 
this procedure 30 times for each ensemble. Curves were obtained 
with populations of neurons segregated from M1, PMd, S1, SMA, and 
PP and for single units and multiunits. 
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Cortical Ensemble Adaptation to Represent Velocity of an 
Artificial Actuator Controlled by a Brain—Machine Interface 
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Monkeys can learn to directly control the movements of an artificial actuator by using a brain-machine interface (BMI) driven by the 
activity of a sample of cortical neurons. Eventually, they can do so without moving their limbs. Neuronal adaptations underlying the 
transition from control of the limb to control of the actuator are poorly understood. Here, we show that rapid modifications in neuronal 
representation of velocity of the hand and actuator occur in multiple cortical areas during the operation of a BMI. Initially, monkeys 
controlled the actuator by moving a hand-held pole. During this period, the BMI was trained to predict the actuator velocity. As the 
monkeys started using their cortical activity to control the actuator, the activity of individual neurons and neuronal populations became 
less representative of the animal’s hand movements while representing the movements of the actuator. As a result of this adaptation, the 
animals could eventually stop moving their hands yet continue to control the actuator. These results show that, during BMI control, 
cortical ensembles represent behaviorally significant motor parameters, even if these are not associated with movements of the animal’s 


own limb. 


Key words: brain-machine interface; motor cortex; macaque monkey; cortical plasticity; motor learning; body schema 


Introduction 

Ever since Evarts (1966) pioneered single-unit recordings from 
the cortex of awake, behaving monkeys, numerous studies have 
scrutinized a fundamental observation: cortical neurons modu- 
late their firing during voluntary movements and thereby encode 
a variety of motor parameters (Georgopoulos et al., 1986, 1992; 
Ashe and Georgopoulos, 1994; Fu et al., 1995; Sergio and Kalaska, 
1998). This encoding incorporates motor, sensory, and cognitive 
signals and appears to be highly complex (Cheney and Fetz, 1980; 
Alexander and Crutcher, 1990; Lebedev et al., 1994; Shen and 
Alexander, 1997; Donchin et al., 1998; Kakei et al., 1999) (for 
review, see Georgopoulos, 2000; Johnson et al., 2001; Andersen 
and Buneo, 2002; Paz et al., 2004). Individual cortical neurons do 
not have a one-to-one relationship to any single motor parameter 
(Ashe and Georgopoulos, 1994; Sergio and Kalaska, 1998; Car- 
mena et al., 2003), and their firing patterns exhibit considerable 
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variability (Lee et al., 1998; Shadlen and Newsome, 1998; Cohen 
and Nicolelis, 2004; Wessberg and Nicolelis, 2004). Precise motor 
control is achieved through the action of large neuronal ensem- 
bles (Wessberg et al., 2000; Carmena et al., 2003). With the ad- 
vent of electrophysiological techniques permitting simultaneous 
recordings from hundreds of neurons in multiple brain areas 
(Nicolelis et al., 1995, 1997, 2003; Kralik et al., 2001; Nicolelis and 
Ribeiro, 2002), detailed information can be collected about en- 
semble encoding. 

Using ensemble recordings, motor parameters can be pre- 
dicted from neuronal activity in real time and used to control a 
brain—machine interface (BMI) (Chapin et al., 1999; Wessberg et 
al., 2000; Nicolelis, 2001; Serruya et al., 2002; Taylor et al., 2002; 
Carmena et al., 2003; Wessberg and Nicolelis, 2004; Patil et al., 
2004). In addition, BMIs can be driven by planning and motiva- 
tion signals extracted from cortical activity (Musallam et al., 
2004). Recently, we implemented a BMI in which cortical ensem- 
ble activity recorded from macaque monkey brain directly con- 
trolled reaching and grasping movements performed by a robotic 
arm (Carmena et al., 2003). The monkeys’ proficiency in operating 
the BMI improved with training, accompanied by changes in neu- 
ronal directional tuning. Based on these observations, we suggested 
that, with prolonged usage, an artificial actuator can be effectively 
incorporated in the internal representation of the subject’s body, 
commonly referred to as “body schema” (Head and Holmes, 1911; 
Gurfinkel et al., 1991; Maravita et al., 2003; Nicolelis, 2003). 

Notwithstanding initial reports that training to operate a BMI 
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evokes plastic changes in the activity patterns of cortical neurons 
(Taylor et al., 2002; Carmena et al., 2003; Musallam et al., 2004), 
a number of questions about cortical representation of artificial 
actuators remain to be addressed. Perhaps the most intriguing 
finding is that neuronal ensembles previously modulated in as- 
sociation with limb movements continue to be modulated after 
the subjects stop moving their limbs, and 
these modulations represent purposeful 
movements of the actuator (Chapin et al., 
1999; Taylor et al., 2002; Carmena et al., 
2003). In the present study, we hypothe- 
sized that neuronal adaptation to repre- 
sent the velocity of an artificial actuator 
instead of the limb can occur even while 
the subjects continue to move their limbs. 
Such adaptations would function to opti- 
mize neuronal representation of new be- 
havioral goals (Todorov and Jordan, 
2002). Accordingly, we examined ensem- 
ble encoding of the hand and actuator 
movements before and after transition to 
BMI control. We discovered changes in 
spatial and temporal representation of 
movements during the BMI control that 
appeared to establish the representation of 
the actuator at the expense of the represen- 5 
tation of the limb. 


A 


Materials and Methods 


Behavioral task and brain—machine interface op- 
erations. Two adult female monkeys (Macaca 
mulatta) were used in this study. All procedures 
conformed to the National Research Council 
Guide for the Care and Use of Laboratory Ani- 
mals (1996) and were approved by the Duke 
University Animal Care and Use Committee. 
The monkeys were trained to pursue a visual 
target on a computer monitor with a cursor, 
initially set to motion using a hand-held pole 
(see Fig. 1A) (Carmena et al., 2003). The pole 
actually controlled the movements of the robot 
arm, which was not visible to the monkeys, and 
the cursor provided visual feedback of the posi- 
tion of the robot (see Fig. 1A). The target ap- 
peared at random locations on the screen, and 
the monkeys had to place the cursor over it to 
obtain a fruit-juice reward [task 1 of Carmena 
et al. (2003)]. Multielectrode arrays consisting 
of 16—64 microwires spaced at 300 zm were 
implanted bilaterally in primary motor cortex 
(M1), dorsal premotor cortex (PMd), supple- 
mentary motor area (SMA), primary somato- 
sensory cortex (S1), and posterior parietal cor- 
tex (PP). Neuronal activity was sampled and 
sorted using a multichannel acquisition proces- 
sor (Plexon, Dallas, TX). 

As the monkeys performed the task by mov- 
ing the pole, a linear model was trained to pre- 
dict hand velocity in real time (mathematical 
description is given below by Eq. 3) (Wessberg 
et al., 2000; Carmena et al., 2003). We call this 
mode of operation pole control. After at least 30 
min of operation in pole control, the model pa- 
rameters were fixed. The robot (and the cursor) 
was disconnected from the pole and was di- 
rectly controlled by brain activity. Robot veloc- 


E1 
5 
£ 
°0 
x 
-5 
0 Time, s 


G1 


Vx, cm/s 


Time, s 


Figure 1. 


Linear 
Model |~e, 


Lebedev et al. ¢ Cortical Adaptation during BMI Control 


ity was set to the predicted values of the model. We name this mode of 
operation brain control, of which we distinguish two types. During brain 
control with hand movements (BCWH), a mode that immediately fol- 
lowed pole control in most experiments, the monkeys continued to move 
the pole. This mode of operation continued for 20-30 min, after which 
the pole was removed from the apparatus. At that time, the monkeys 
stopped moving their arms, as confirmed by EMG recordings (Carmena 
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Experimental design and modes of operation. A, Experimental apparatus. The monkey was seated in front of a 
computer monitor on which visual stimuli were shown. It had to pursue a visual target (large circle) with a cursor (small circle). The 
monkey controlled the cursor by moving a hand-held pole (pole control). The pole actually controlled a robotic arm invisible to the 
monkey, and the cursor position on the screen reflected the robot's position. A linear model was trained to predict hand/robot 
velocity from neuronal ensemble activity recorded from the monkey's cortex. Then, the pole was disconnected, and the robot was 
directly controlled by the model's output (brain control). B, Schematics of movement trajectory, instantaneous velocity, and 
neuronal discharges preceding or succeeding an IVM. Neuronal rates were estimated using 100 ms bins placed at different lags 
relative to the IVM. C, Representative traces of the hand (black) and the robot (red) during pole control. D, Hand and robot traces 
during brain control. £7, £2, Time-dependent traces of the hand and robot position during pole control. F7, F2, Time-dependent 
position traces during brain control. G7, G2, Hand and robot velocity during pole control. E7, E2, Velocity during brain control. 
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et al., 2003), but continued to operate the BMI. This mode of operation is 
termed brain control without hand movements (BCWOH). BCWOH 
was first attempted after the monkeys were trained in BCWH for 1 week; 
the monkeys were able to control the actuator in this mode of operation 
and obtain rewards on the very first attempt. Each recording session 
continued for 60-90 min, until the monkeys had had enough juice and 
became unmotivated to perform the task. In some sessions, to obtain 
longer records of BCWOH, the pole was removed immediately after pole 
control. 

Tuning to velocity during pole and brain control. Significant correlation 
between neuronal firing rate and movement velocity has been docu- 
mented in previous studies of M1 (Georgopoulos et al., 1988; Ashe and 
Georgopoulos, 1994; Moran and Schwartz, 1999; Carmena et al., 2003) 
and other cortical areas (Carmena et al., 2003; Averbeck et al., 2005). This 
finding was key for the experiment reported here, because the BMI relied 
on predictions of velocity. The linear model predicted velocity in real 
time, and the velocity of the robot was set to the output of the model. In 
the present analysis, we describe the relationship between neuronal firing 
rate and movement by examining the firing rate of each neuron as a 
function of velocity and of the time lag between the firing rate and in- 
stantaneous velocity measurement (IVM) (see Fig. 1B). IVMs were sam- 
pled every 100 ms. Neuronal activity was binned in 100 ms nonoverlap- 
ping windows. Binned firing rates were then normalized by subtracting 
the mean and dividing by the SD. Lags in the range of 1 s before to 1 s after 
the IVM were examined. For each lag 7, neuronal firing rate n was fitted 
to velocity using the following multiple linear regression: 


(1) 


where t is time, n(t + 7) is neuronal firing rate at time 7 from velocity 
measurement (negative T corresponds to activity preceding velocity mea- 
surement, and positive 7 activity succeeding velocity measurement), 
V,{t) and V,(t) are x- and y-components of velocity, a(), b(7), and c(7) 


n(t + 7) = a(t)V,(t) + B(7)V,(t) + c(7) + a(t, 7), 
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Figure2. Frequency distributions of velocity (V, and V)) for different operations for monkey 1 (A—D) and monkey 2 (E-H).A, 
E, Pole control, hand velocity. B, F, Brain control with hand movements, hand velocity. C, G, Brain control with hand movements, 


robot velocity. D, H, Brain control without hand movements, robot velocity. 


ses 


J. Neurosci, May 11, 2005 + 25(19):4681— 4693 » 4683 


are regression coefficients, and e(t, T) is the residual error. We character- 
ized the degree of correlation between the neuronal firing rate and the 
velocity as the velocity tuning index (VTI), which was calculated as the 
square root of R? for the regression (Eq. 1). This measure describes 
correlation between neuronal firing rate and movement velocity (both 
direction and amplitude taken into account) and represents it in normal- 
ized form. The better the neuronal firing rate tracked movement velocity, 
the higher was the VTI. The VTI did not depend on the absolute value of 
firing rate (e.g., highly active versus less active neurons) because of nor- 
malization. Preferred direction (PD; i.e., movement direction for which 
the neuron fired the most) was calculated as PD(tT) = arctan b(t)/a(7). 

To examine whether the PDs of individual neurons in the population 
were preserved after the monkey started to perform in a new mode of 
operation (e.g., pole control vs BCWH), we used a correspondence index 
defined as follows: 


90° — ja - Bl 
C= 90° > (2) 
where statistical values are calculated for the neuronal samples of the PDs 
(measured in degrees) before and after the transition, a and B, respec- 
tively, the bar denotes the mean, and 90° is the mathematical expectation of 
a— f| in case they are uncorrelated. C was scaled from 0 to 1. Values of C 
approaching zero indicate no correspondence between the PDs, whereas 
values of C approaching unity indicate an exact correspondence. 

To examine the contribution of correlation between neurons and tun- 
ing parameters, we conducted a shuffle test that destroyed the correlation 
between neurons by shifting spike trains of different neurons with respect 
to each other by a random interval in the range of 0 to 200 s. This 
procedure preserved the sequence of spikes of each neuron but removed 
correlations between the neurons. After shuffling, robot movements 
were recalculated using the same model with the fixed weights that had 
been used for the real-time brain control. The VTI of each individual 
neuron for shuffled data was a measure of tun- 
ing that would be expected simply because that 
neuron had a weighted contribution to the ro- 
bot control signal. Higher values of VTIs for 
unshuffled data indicated that correlated firing 
between neurons improved tuning characteris- 
tics of individual neurons (i.e., neuronal sub- 
groups acted as “teams” producing movements 
in particular directions). 

A separate analysis tested velocity tuning 
properties for instances in which the monkeys’ 
hands or the robotic actuator moved in 
straight-line trajectories. Epochs of straight- 
line movements of at least 300 ms duration 
were selected. Firing rates were measured using 
a 300 ms window placed at different time lags 
relative to the selected epochs. Regression of 
Equation 1 was used to determine the VTI. In ad- 
dition, directional tuning depth was calculated 
(Carmena et al., 2003). Directional tuning depth 
was equal to the difference in the average firing 
rate between the directions of maximum and 
minimum firing, divided by the SD of the firing 
rate. 

Off-line predictions of hand velocity. Predic- 
tions of velocity were made from a weighted 
linear combination of neuronal firing rates us- 
ing the following: 


D scwo, Robot 


H_ scwo, Robot 


n 


V.(t) =b+ >) w(a)n(t+ 7) + a(t), 


Tm 


(3) 
where n(t + 7) is a vector of neuronal firing 
rates, at time ft and time lag T (negative lags cor- 
respond to past events), V,, is the x-component 
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of velocity at time t, w(7) is a vector of weights 
for each neuron at time lag 7, b is the A 
y-intercept, and e(t) is the residual error. An 
analogous equation was used to estimate V,. 
These equations were solved using linear least- 
squares regression. Ten time lags preceding ve- 
locity measurement with a bin size of 100 ms 
were used for predictions. Ina separate analysis, 
estimations of velocity were calculated using a 
single 100 ms bin placed at different lags relative 
to velocity. Estimation quality was quantified 
by the correlation coefficient (R). 

Random neuron dropping was implemented 
as follows: 10 min of neuronal population data 
were used to fit the model and find the weights, 
w(T). These weights were then used to make 
predictions of velocity on a different 10 min 
period. A single neuron was then randomly re- 
moved from the population, the model re- 
trained, and new predictions generated on the 
same data. This process was repeated until no 
neurons remained. Average R as a function of 
number of neurons was obtained by repeating 
this procedure 100 times. 


Results 
Hand and robot velocity during xt 
different modes of operation 

The linear model was trained during pole 
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control. However, after the mode of oper- 
ation was switched to brain control, robot 
movements did not precisely match move- 
ments of the hand. This is illustrated by 
Figure 1C—H, which shows representative 
traces of the hand and the robot during 
pole and brain control. During pole con- 
trol, movements of the hand and the robot 
were in close correspondence (Fig. 1C), 
which can also be seen in time-dependent 
plots of the x- and y-components of posi- 
tion (Fig. 1£1,E2) and velocity (Fig. 
1G1,G2). During BCWH, the hand and the 
robot moved differently (Fig. 1D), albeit 
in a correlated manner (Fig. 1 F,H). 
Figure 2 shows the frequency distribu- 
tion of hand and robot velocities (V,, and 
V,) during different modes of operation in 
two monkeys. The correlation coefficient 
between the robot and hand velocities was 
0.995 + 0.001 (mean + SE) during pole 
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Figure 3. Velocity tuning in M1 neuron tuned during pole control and brain control. A, Color plots of the firing rate of the 
neuron (color coded; key, bottom left) as a function of V, and y, (key, bottom left) for different lags with respect to IVM, different 
modes of operation (pole control and brain control with and without hand movements), and different velocity parameters (hand 
or robot movements). B, Velocity tuning index as a function of lag for different types of operation (color coded; key on top). C, 
Preferred direction as a function of lag (color coded; key on top). deg, Degrees. 


control versus 0.61 + 0.02 during BCWH 

in monkey 1 and 0.980 + 0.001 versus 

0.50 + 0.04 in monkey 2. The difference between these values was 
highly significant statistically (p < 0.001; Wilcoxon rank-sum 
test). Ranges of hand velocities were similar during pole (Fig. 
2A,E) and brain (Fig. 2B, F) control. However, a comparison of 
robot velocities (Fig. 2C,G) with the hand velocities that accom- 
panied them during brain control (Fig. 2B, F) showed that lower 
robot velocities were less frequent than those of the hand. Abso- 
lute velocity of the hand (5.65 + 0.03 cm/s in monkey 1 and 
4.51 + 0.04 cm/s in monkey 2) was statistically different ( p < 0.001; 
Wilcoxon rank-sum test) from absolute velocity of the robot 
(6.93 + 0.03 and 7.00 + 0.03 for monkeys 1 and 2, respectively). 
Thus, the monkey’s hand often moved slowly, but the robot moved 
faster, driven by neuronal ensemble activity. Ranges of robot veloc- 


ities were similar during BCWH (Fig. 2C,G) and BCWOH (Fig. 
2D,H). 


Neuronal tuning during different modes of operation 

We analyzed neuronal data collected in nine daily recording ses- 
sions in monkey 1 and in 20 sessions in monkey 2. Analyzed 
neuronal samples on average consisted of 177 units per recording 
session in monkey 1 and 72 units in monkey 2. Examination of 
neuronal rates at different time lags relative to IVM (Fig. 1B) 
showed modulations correlated with velocity magnitude and di- 
rection (Figs. 3-5). We refer to correlation between neuronal rate 
and velocity (Georgopoulos et al., 1988; Ashe and Georgopoulos, 
1994; Moran and Schwartz, 1999; Carmena et al., 2003) as veloc- 


Lebedev et al. ¢ Cortical Adaptation during BMI Control 


Lag = -400 ms 

-300 ms 

-200 ms 
-100 ms 
Oms 


A 


o 
> x3 
oa =——s Ss 
-14 0 14 -0.9 1.1 Hs 
V,, m/s Normalized Rate Sixt 
oN RS 2 
Ns 
RS 
Qe 
Pole Control 


w 


Cc 


Instantaneous 
Velocity 
Measurement (IVM) 


+100 ms 
+200 ms 


Brain Control with Hand Movements, Relative to Hand 
Brain Control with Hand Movements, Relative to Robot 
Brain Control without Hand Movements, Relative to Robot 


J. Neurosci., May 11, 2005 + 25(19):4681— 4693 * 4685 


zero velocity; x- and y-axes correspond to 
x- and y-components of velocity, respec- 
tively. These color plots are useful to eval- 
uate the velocity tuning of a neuron at a 
glance. They contain information about 
both directional tuning of the neurons (di- 
rection is represented by a vector originat- 
ing in the center of the plot) and their sen- 
sitivity to velocity magnitude (velocity 
magnitude is measured as the distance 
from the center of the plot). The color 
plots were constructed for different lags 
relative to the IVM and for different 
modes of operation (compare Figs. 3-5). 
Two sets of color-plots were calculated for 
BCWH, because during this mode of op- 
eration, hand movements did not match 
robot movements. One set relates firing 
rate to velocity of hand movements (Fig. 
3A, second row from top). The other was 
calculated relative to robot velocity (Fig. 
3A, third row from top). The color plots 
oo for BCWOH were calculated relative to ro- 
bot velocity (Fig. 3A, bottom row). In ad- 
dition, VTIs (Fig. 3B) and PDs (Fig. 3C) 
were plotted as a function of lag for differ- 
ent modes of operation. 

Individual neurons exhibited a diver- 
sity of tuning patterns that depended on 
the mode of operation. Figures 3—5 illus- 
trate three representative neurons re- 
corded in M1. Their tuning characteristics 
were averaged over five consecutive re- 
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Figure 4. An M1 neuron that was tuned to movement velocity only ifthe monkey’s hand moved. Conventions are asin Figure curve). During BCWH, tuning relative to 
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ity tuning. The degree of velocity tuning was quantified as the VTI 
(see Materials and Methods). This normalized index measured 
the dependency of neuronal firing rate on both direction and 
magnitude of velocity. Because of normalization, the VTI was not 
biased toward highly active neurons. Neurons tuned before the 
IVM are more likely to have a causal relationship to movements, 
for example by projecting to spinal motoneurons. Neurons tuned 
after the IVM may reflect sensory feedback. Pre-IVM and post- 
IVM tuning can also be related to movement patterns, such as 
acceleration and deceleration (Sergio and Kalaska, 1998; Johnson 
and Ebner, 2000; Todorov, 2000). The dependency of firing rate 
on x- and y-components of velocity is depicted in color plots (Fig. 
3A) (Johnson and Ebner, 2000; Lebedev et al., 2003; Paninski et 
al., 2004). The color encodes firing rate (red, high rate; blue, low 
rate). The center of each color plot represents the firing rate for 


hand movements (Fig. 3B, red curve) was 
significantly diminished. However, tuning 
relative to robot movements (Fig. 3B, black curve) was compara- 
ble with tuning during pole control (no statistical difference). 
Tuning during BCWOH was measured relative to robot move- 
ments. It was significantly lower than tuning relative to robot 
movements during BCWH but higher than tuning relative to 
hand movements during BCWH. The PD of this neuron was 
stable and similar for different modes of operation for the lags 
preceding the IVM (Fig. 3C). For the points after the IVM, the PD 
rotated for all modes of operation except BCWOH. Thus, after 
transition to brain control, this neuron retained many features of 
its original velocity tuning. However, it became less tuned to 
hand movements. 
Figure 4 illustrates a neuron whose tuning mostly reflected 
hand movements. Changes in the VTI during different modes of 
operation were statistically significant ( p < 0.001; Kruskal—Wal- 
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lis ANOVA; a = 0.05 for Tukey’s multiple 
comparison). This neuron was tuned dur- 
ing pole control and BCWH. However, ve- 
locity tuning vanished during BCWOH 
(Fig. 4A,B). Similar to the neuron illus- 
trated in Figure 3, the highest VTIs of this 
neuron were observed for lags preceding 
the IVM, which may have been related to 
the causal relationship between the neuro- 
nal activity and hand movements. How- 
ever, this neuron was also tuned after the 
IVM, especially during pole control. The 
PD depended on the lag relative to the 
IVM. It rotated counterclockwise (Fig. 4C) 
and completely reversed for lags succeed- 
ing the IVM. Comparable PD reversals 
have been reported previously in arm move- 
ment tasks and attributed to a representation 
of accelerative and decelerative forces (Ser- 
gio and Kalaska, 1998; Todorov, 2000). 
Note, in addition, that post-IVM tuning was 
markedly decreased when tuning was as- 
sessed relative to robot movements during 


brain control. Prevalence of pre-IVM tuning 14 0 14 
was also observed for the whole neuronal en- V,, cm/s 
semble (see below). Because of the strong de- 

pendency of tuning on the presence of hand 

movements, this neuron is likely to have 


been critically involved in generation of de- 
scending motor commands (e.g., pyramidal 
tract neuron). 

Pairwise comparison of pole control 
with different types of brain control 
(which was somewhat different from 
grouping the data from different sessions 
together, because different types of brain 
control were not tested in every session) 
showed that in the majority of neurons 
(77 = 2%; mean + SE), the peak VTI for 
hand movements decreased after the tran- 
sition to BCWH. During BCWH, the peak 
VTI relative to robot velocity exceeded the 
peak VTI relative to hand velocity in 84 + 
2% of neurons. Each of these differences in 
proportions was highly significant statisti- 
cally (p < 0.001; Wilcoxon signed rank 
test). Importantly, the peak VTI of a sub- 
stantial number of neurons (38 + 2%) was 
higher during BCWOH than during pole 
control. A neuron of the latter type (Fig. 5) 
had a peculiar tuning pattern: it was much better tuned during 
BCWOH than other conditions (p < 0.001; Kruskal-Wallis 
ANOVA; a = 0.05 for Tukey’s multiple comparison). 

Population averages for all recording sessions confirmed that 
neurons typically became less tuned to hand movements and 
better tuned to robot movements (Table 1). Population aver- 
ages of peak VTIs were 0.074 + 0.002 for pole control, 0.056 + 
0.002 for BCWH relative to hand, 0.077 + 0.003 for BCWH 
relative to robot, and 0.070 + 0.001 for BCWOH relative to 
robot (mean + SE) in monkey 1 and 0.104 + 0.005, 0.069 + 
0.004, 0.118 + 0.002, and 0.126 + 0.007 in monkey 2. Changes 
in the peak VTI were statistically significant (p < 0.001; 
Kruskal-Wallis ANOVA). Pairwise comparison (Tukey’s mul- 
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tiple comparison; a = 0.05) showed that the peak VTI during 
BCWH relative to hand was significantly less than the peak 
VTI during pole control and BCWH and BCWOH relative to 
robot. This decrease in tuning to hand movements and better 
tuning to the actuator instead was observed in the majority of 
sampled areas. Statistical comparison of peak VTIs for indi- 
vidual cortical areas is presented in Table 1. 


Tuning patterns for the whole ensemble 

Color plots of Figure 6 show VTIs as a function of lag for the 
whole neuronal ensemble. They present VTI curves (Figs. 3B, 4B, 
5B) in color-coded format, stacked one above the other, so that 
each line corresponds to a neuron. It can be seen that for the 
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Table 1. Velocity tuning index (mean + SE) by cortical area and condition 
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Pole control (a) 


BCWH, relative to hand (b) 


BCWH, relative to robot (c) 


BCWOH, relative to robot (d) 


BCWOH, shuffled data (e) 


Monkey 1 

PMd (n = 62) 

M1 (n = 54) 

$1 (n = 37) 

SMA (n = 18) 

M1 ipsilateral (n = 6) 

All neurons (n = 177) 
Monkey 2 

PP (n = 52) 

M1 (n = 15) 

SMA (n = 5) 

All neurons (n = 72) 


0.047 + 0.002 ¢, d 
0.127 + 0.005 b, d, e 
0.059 + 0.003 b, d, e 
0.055 + 0.006 

0.035 + 0.002 d 

0.074 + 0.002 b, c,d, e 


0.104 + 0.005 b, ¢, d 
0.080 + 0.004 b, c, d 
0.078 + 0.005 b 

0.097 = 0.004 b, c, d 


0.046 + 0.003 ¢, d 
0.089 = 0.004 a, c, e 
0.036 + 0.002 a, c,d, e 
0.049 + 0.008 d 

0.026 + 0.003 d 

0.056 + 0.002 a, ¢, d 


0.069 + 0.004 a,c, d,e 
0.042 + 0.002 a, c,d, e 
0.040 + 0.002 a,c, d,e 
0.061 + 0.003 a, c,d, e 


0.063 + 0.004 a, b, e 
0.116 = 0.006 b, e 
0.054 = 0.003 b, d 
0.063 + 0.010 

0.037 = 0.002 

0.077 + 0.003 a, b, e 


0.118 + 0.005 a, b, e 
0.102 + 0.002 a, b, e 
0.108 + 0.005 b 

0.114 + 0.004 a, b, e 


0.060 + 0.002 a, b, e 
0.096 + 0.003 a, e 
0.060 + 0.002 b, ¢, 
0.055 + 0.004 b 
0.044 + 0.002 a, b, e 
0.070 + 0.001 a, b, e 


0.126 + 0.007 a, b, e 
0.125 = 0.005 a, b, e 
0.126 + 0.006 b 

0.126 + 0.005 a, b, e 


0.045 + 0.001 ¢, d 
0.071 + 0.003 a, b,c, d 
0.044 + 0.001 a, b, d 
0.046 + 0.003 

0.034 + 0.002 d 

0.052 + 0.001 a, c,d 


0.082 + 0.007 b, ¢, d 
0.065 + 0.002 b, ¢, d 
0.094 + 0.009 b 

0.079 + 0.005 b, ¢, d 


Numbers represent mean + SE and the columns for which Tukey's multiple comparison test indicated significant difference (p < 0.05). 


ensemble as a whole, tuning to hand movements decreased dur- 
ing BCWH (Fig. 6A2) compared with tuning during pole control 
(Fig. 6A1) and tuning to robot movements during BCWH (Figs. 
6A3, 4). Individual neurons exhibited a diversity of VTI changes. 
Averaging VTIs for the whole ensemble (Fig. 6B) confirmed a 
decrease in tuning to hand movements during BCWH (Fig. 6B, 
compare blue and red curves). VTIs for robot movements (Fig. 
6B, black and green curves) were clearly higher before than after 
the IVM. This occurred for two reasons. First, only the bins pre- 
ceding the IVM contributed to the on-line predictions of velocity. 
Second, as a result of the delay of the robotic system (~100 ms), 
the curves were shifted to the left. VTIs for hand movements 
during pole and brain control peaked closer to VM compared 
with robot movements. Average VTIs for the whole ensemble 
were similar to those for M1 (Fig. 6 B,C), because in the sampled 
neuronal population, M1 neurons were tuned most strongly. 

The occurrence of tuning of the neuronal ensemble to robot 
velocity during BCWOH is not surprising, because the weighted 
sum of neuronal firing rates was made to control the robot. How- 
ever, neuronal tuning to robot movements during BCWOH was 
not a simple consequence of using the linear model (as it would 
be if just one neuron controlled the BMI). Rather, correlated 
firing of the whole ensemble played a role in the tuning properties 
of individual neurons. This was shown using the shuffle test. The 
average VT] decreased after the correlation between the neurons 
was destroyed by the shuffling procedure (see Materials and 
Methods), as seen in Figure 6, B and C (dotted green lines vs solid 
green lines) and Table 1. This decrease was statistically significant 
for both monkeys ( p < 0.001; Wilcoxon signed rank test; means 
and SDs are presented in Table 1). Thus, tuning of any individual 
neuron was not related only to the weights assigned to that neuron 
by the linear model but also depended on the firing of the rest of the 
ensemble. We have shown previously (Carmena et al., 2003) that 
firing of individual neurons was correlated, and this correlation in- 
creased in brain control. Because of the enhancement in correlated 
activity, VTIs of individual neurons increased as well. 

Preferred directions plotted as a function of lag relative to the 
IVM (Fig. 7A) showed that, generally, there was no fixed PD for 
each neuron. Rather, PDs rotated with lag changes similarly to 
the examples in Figure 3 and 4. This rotation is consistent with 
previous results of Sergio and Kalaska (1998), Todorov (2000), 
and Johnson and Ebner (2000). After the mode of operation was 
switched to brain control, if the monkeys continued to move 
their arms, the pattern of PDs generally resembled that of pole 
control (Fig. 7A1—A3). Correspondence index analysis indicated 
that the best correspondence between the PDs measured with 


respect to hand movements during pole control and BCWH oc- 
curred at the time of the IVM (Fig. 7B, red line), whereas PDs 
measured with respect to robot movements were in best corre- 
spondence to pole control ~100 ms before the IVM (Fig. 7B, 
black line). This is in agreement with average VTI curves (Fig. 
6B). Statistical analysis of peak values of correspondence indices 
showed better correspondence between the PDs during pole con- 
trol and BCWH measured with respect to robot movements 
compared with the values measured with respect to hand move- 
ments, which was statistically significant for monkey 1 (0.38 + 
0.01 vs 0.33 + 0.01; mean + SE; p < 0.05; Wilcoxon signed rank 
test) and did not reach statistical significance for monkey 2 
(0.66 + 0.03 vs 0.59 + 0.04; p > 0.05). The PDs during BCOWOH 
(Fig. 7A4) were confined to a narrower angular range and rotated 
much less with changes in the lag than those observed during pole 
control (Fig. 7A1) and BCWH (Fig. 7A2,A3) (Carmena et al., 2003). 
Correspondence index analysis, however, indicated a relatively good 
match between the PDs during BCWOH and pole control for the 
100 ms lag preceding the IVM (Fig. 7B, green line). Peak values of the 
correspondence between the PDs during BCWOH and pole control 
for monkey 1 (0.24 + 0.01) were significantly less than the corre- 
spondence index between pole control and BCWH ( p < 0.05; Wil- 
coxon signed rank test), and the values for monkey 2 (0.52 + 0.05) 
were not significantly different ( p > 0.05). 


Neuronal tuning for selected hand movements 

Changes in neuronal tuning to hand movements that occurred 
after the monkeys started to perform in brain control could be 
related, among other factors, to differences in the pattern of hand 
movements. To test whether these changes would still be present 
if the same hand movements were compared, we selected epochs 
during which the monkeys’ hands or the robot moved in straight 
lines for at least 300 ms (Fig. 8A). For these epochs, we con- 
structed classical tuning curves, which represented average firing 
rate as a function of movement direction (Fig. 8B). Directional 
tuning depth (Fig. 8C) and VTI (Fig. 8 D), calculated as functions 
of the lag between the 300 ms window in which firing rate was 
measured, showed the same effect as demonstrated with other 
methods: both directional tuning depth and VTI with respect to 
hand movements were less during BCWH than during pole control. 
Average peak tuning depth was 0.55 + 0.01 and 0.71 + 0.01 in 
monkeys 1 and 2, respectively, during pole control, and it was 0.44 + 
0.01 and 0.55 + 0.01 during brain control. Average peak VTI was 
0.203 = 0.005 and 0.277 + 0.005 (monkeys 1 and 2) during pole 
control versus 0.161 + 0.004 and 0.207 + 0.005 during brain con- 
trol. These differences between tuning to hand movements in pole 
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control and brain control were statistically A 
significant (p < 0.001; Wilcoxon signed 


rank test). P Hand 
an 


Off-line predictions of hand velocity mee 


during pole and brain control 

Similar to the decrease in VTIs, the quality 
of off-line predictions of hand velocity de- 
creased during BCWH (Fig. 9A). In this 
analysis, the linear model was trained to 
predict hand velocity using either pole- 
control or brain-control data as training 
epochs. Ten minute (Fig. 9A) or 5 min 
(Table 2) data segments were used for 
training. Predictions were then tested ona 
new 10 or 5 min epoch (splitting the data 
into 5 min epochs was useful for statistical 
analysis, although prediction quality was 
less compared with 10 min epochs, which 
was also used for training the model dur- 
ing online predictions). Prediction quality 
dropped during brain control (Fig. 9A). 
This decrease occurred quite abruptly after 
the operation was switched to brain con- 
trol (Fig. 9B). Average R values for predic- 
tions of hand velocity using 5 min seg- 
ments of data during pole control were 0 
0.64 + 0.02 and 0.48 + 0.01 for monkey 1 
and 2, respectively, whereas they were 
0.40 + 0.01 and 0.30 + 0.01 for predic- 
tions of hand velocity made during BCWH 
(Table 2). This change in R values was sta- 
tistically significant (p < 0.001; Kruskal— 
Wallis ANOVA; a = 0.05 for Tukey’s mul- 
tiple comparison). 

To inspect changes in contribution to 
velocity prediction for different lags be- 
tween the velocity and neuronal activity, 
velocity predictions were performed using 
a single 100 ms sampling window. The 
window was placed at different lags with 
respect to the IVM. This analysis showed a -1 
decrease in prediction for all tested lags 
(Fig. 9C). 

It is important to note that the same 
drop in prediction quality for hand veloc- 
ity as we show here in the off-line analyses 
occurred during the experimental sessions 
as well after the mode of operation was 
switched to brain control. However, such 
deterioration in prediction quality only 
meant that the model became less predic- 
tive of the velocity of the hand. This decrease in predictions of hand 
movement did not impede the behavioral performance, which was 
determined by the robot movements rather than hand movements. 
Behavioral performance improved with training in brain control 
(Carmena et al., 2003), suggesting that cortical representation of the 
robot was optimized at the expense of representation of the animal’s 
own limb. 
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Discussion 
The principal finding of this study is that once cortical ensemble 
activity is switched to represent the movements of the artificial 
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control without hand movements is shown by the green dotted line. Averages are shown for the whole ensemble. C, Average VTls 
as a function of lag for M1 only. 


actuator, it is less representative of the movements of the animal’s 
own limb. Although the original rate of the sample of recorded 
cortical neurons was initially modulated in association with hand 
movements, after these neurons became involved in brain con- 
trol, they became less correlated with hand movements. More- 
over, neuronal tuning to the movements of the actuator was en- 
hanced because of increased correlation between the neurons. 
This result accords with the vast literature showing that neuronal 
modulations in cortical motor areas do not exclusively reflect 
limb movements and that cognitive signals are abundant in these 
areas (for review, see Wise et al., 1997; Georgopoulos, 2000; John- 
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Neurophysiological studies have 
shown that modulations in cortical areas 
that occur during movement planning and 
execution often reflect abstract parameters 

ips of movements rather than low-level de- 
tails. Cortical neurons can encode move- 
ment direction regardless of muscle pat- 
terns (Kakei et al., 1999, 2001), represent 
target of movement and movement of the 
S1 hand-controlled visual marker toward it 
rather than actual hand trajectory (Alex- 
ander and Crutcher, 1990; Shen and Alex- 
ander, 1997a,b; Ochiai et al., 2002), encode 
multiple spatial variables (Lebedev et al., 
2004; Cisek and Kalaska, 2005), reflect ori- 
M1 entation of selective spatial attention (Leb- 
edev and Wise, 2001), and represent mis- 
perceived movements of visual targets 
(Lebedev et al., 2001; Lebedev and Wise, 
2002; Schwartz et al., 2004). In addition, 
frontal cortex neurons can flexibly adapt 
to new visuomotor (Wise et al., 1998; Paz 
et al., 2003; Paz and Vaadia, 2004) and dy- 
namic (Li et al., 2001; Padoa-Schioppa et 
al., 2004) conditions. Based on the results 
presented here, we suggest that neuronal 
mechanisms underlying these adaptive 
properties can also subserve cortical ensem- 
ble adaptation during the BMI operation. 
Operating a BMI is not unlike using 


PMd 


3 0.4 During Pole Control mechanical tools, and the idea that body 
fC Compared to schema can extend along a wielded tool 
9 BCWH, Hand was formulated almost 100 years ago 
5 270 (Head and Holmes, 1911). This suggestion 
E BCWH. Robot was substantiated much later by neuro- 
fe) : Preferred : : : : 
a Direction, deg physiological data showing that cortical 
2 BCWOH, Robot : neurons responsive to both somatosen- 
8 0, IVM 1 sory and visual stimuli extended their re- 
Lag, s ceptive fields along a rake that a monkey 
; used to retrieve distant objects (Iriki et al., 
Figure 7. Changes in preferred directions for the whole ensemble. A, PD as a function of lag relative to IVM for the ensemble 1996). Effects consistent with the idea of 


recorded in monkey 1. Each row corresponds to a neuron. PDs are shown for pole control (1), brain control with hand movements, 
relative to hand (2) and to robot (3), and brain control without hand movements, relative to robot (4). ips, Ipsilateral. B, Corre- 
spondence index describing the similarity between PD distribution during pole control and brain control. deg, Degrees. 


son et al., 2001; Andersen and Buneo, 2002; Paz et al., 2004). For 
instance, neuroimaging studies in human subjects have shown an 
activation of cortical motor areas in the absence of limb move- 
ments (e.g., when subjects imagine that they produce move- 
ments) (Leonardo et al., 1995; Porro et al., 1996; Ehrsson et al., 
2003) or when the illusion of limb movement is induced by mus- 
cle vibration (Naito et al., 1999). Moreover, Graziano et al. (Gra- 
ziano, 1999; Graziano et al., 2000) demonstrated that if the vision 
of the monkey’s own arm was occluded, PMd and PP neurons 
reflected the position of a visible, realistic false arm, but at the 
same time, they were not sensitive to unrealistic substitutes for 
the arm. A recent neuroimaging study confirmed this result by dem- 
onstrating similar activations that occurred in human premotor cor- 
tex when the subjects experienced a rubber hand illusion (Ehrsson et 
al., 2004). Thus, cortical representation of the limb is highly flexible 
and susceptible to illusions and mental imagery. 


remapping body schema after tool usage 
have been also observed in human psycho- 
physics experiments (for review, see Mara- 
vita et al., 2003). A recent neuroimaging 
study (Maruishi et al., 2004) reported spe- 
cific activations of the right ventral premo- 
tor cortex during manipulation of a myoelectric prosthetic hand. 
The authors interpreted this result as the recognition by the brain 
of the prosthetic hand as a “high performance alternative to a real 
hand.” Perhaps a long-term operation of a BMI]-controlled actua- 
tor would lead to even more vivid perception of the actuator 
being a body part rather than merely a tool or an alternative to the 
real hand. In support of this suggestion, perceived voluntary 
movements of phantom limbs in amputees were associated with 
primary sensorimotor cortex activation (Roux et al., 2003). 

When interpreting neuronal patterns during BMI control as 
reflecting a new representation of the artificial actuator, however, 
one should be careful, because the BMI design effectively makes 
any modulation of neuronal activity translate into the move- 
ments of the actuator. One piece of evidence that the system 
adapts to control the actuator comes from the nonrandomness of 
the actuator movements and behavioral improvements as the 
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tories of the hand selected for eight movement directions during pole control and brain control 
with hand movements. B, Directional tuning curves calculated for the traces above by averaging 
firing rates for different movement directions within a 300 ms window leading the movement 
by 100 ms. Each horizontal line represents a curve for a particular neuron. ips, Ipsilateral; deg, 
degrees. C, Directional tuning depth as a function of lag between the firing rate window and 
movement trace. D, Velocity tuning index. 


monkey trains (Taylor et al., 2002; Carmena et al., 2003). The 
second piece of evidence is coordinated activity of the neurons in 
the ensemble. Extending our previous demonstration of in- 
creased correlation between the neurons during brain control 
(Carmena et al., 2003), we showed here using the shuffle test that 
neuronal correlation enhanced tuning to robot velocity in indi- 
vidual neurons. Moreover, tuning of the same neurons to hand 
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velocity decreased, and this decrease occurred even for the relatively 
stereotypical subset of hand movements having a straight-line tra- 
jectory (Fig. 8). Together, these observations suggest that the neuro- 
nal ensemble adapted to represent the actuator at the expense of the 
representation of the animal’s own limb. 

Tuning patterns in the entire ensemble (Fig. 5A) during pole 
control and brain control with and without hand movements 
showed that, in general, most of the neurons that encoded move- 
ment velocity during pole control continued to do so during the 
BMI operation. How then could the monkeys operate the BMI 
and at the same time inhibit movements of their hands? We sug- 
gest that subtle rather than dramatic differences in the way the 
activity of individual neurons was combined at the ensemble level 
determined this novel behavioral outcome. For example, al- 
though the overall modulation pattern was similar across behav- 
ioral states, some neurons (Fig. 4) did not modulate when the 
monkeys’ hands stopped moving, whereas others (Fig. 3) were 
strongly modulated. It should be noted that the neuronal ensem- 
ble that was recorded and used to control the BMI constituted 
only a tiny part of the entire network normally involved in con- 
trolling arm movements. It is possible that these selected neurons 
were preferentially modulated to control the actuator during BC- 
WOH, whereas the ones that were not recorded from continued 
to operate as normally. Alternatively, the ensemble controlling 
the BMI could be assisted by a neuronal network to which it was 
connected. Additional studies will be needed to resolve between 
these possibilities. 

Our finding that after learning to control an artificial actuator 
using a BMI, neuronal ensembles represented the actuator veloc- 
ity better than the velocity of the animal’s own arm appears to 
support the theory of optimal feedback control (Todorov and 
Jordan, 2002; Scott, 2004). This theory describes the motor sys- 
tem as a stochastic feedback controller that optimizes only those 
motor parameters that are necessary to achieve task goals. In our 
study, the task goals were achieved by the robot. Thus, in terms of 
optimal feedback control theory, cortical ensembles should adapt 
to represent the goal-related variable although it is related to the 
robot instead of the limb. 

The present observations also suggest that the quality of pre- 
dictions in open-loop mode that computational neuroscientists 
are striving to achieve may be relatively unimportant for closed- 
loop BMI control. Once the neuronal ensemble starts to control a 
BMI, imperfections in the model may be compensated by neuro- 
nal adaptations that improve control of the actuator movements. 
Whereas they improve the BMI performance, these adaptations 
may actually worsen the predictions of hand movements, the 
effect that we observed here. These considerations concur with 
the results of Taylor et al. (2002), who showed differences in 
tuning properties of M1 neurons during hand movements and 
during a BMI control that used a coadaptive algorithm for adjust- 
ing the model weights in the absence of hand movements. 

To clarify the dynamics of cortical plasticity associated with 
BMI control and to elucidate the exact mechanism of cortical 
adaptation to represent an artificial actuator will require addi- 
tional studies. It is very likely that the features of such adaptations 
will depend on concrete requirements for the BMI operations. 
For example, it is possible that if the subject is required to sub- 
serve the BMI control with limb movements, the system may 
come to an optimal solution in which both the limb and the 
actuator are represented. Alternatively, if limb movement is not 
required, the system may gradually evolve to a different state 
(Taylor et al., 2002; Carmena et al., 2003). It is also possible that 
certain neuronal adaptations could work as temporary solutions 


297 


Lebedev et al. ¢ Cortical Adaptation during BMI Control 


Predictions of Hand Velocity during 
mm Pole Control 
vst BCWH, Model Trained in Pole Control 


= = = BCWH, Model Trained in Brain Control 


0 
10 20 30 40 -1 
Time, min 


0 
200 0 


0 
0 100 
Number of Neurons 


Figure 9. 


Table 2. Prediction of hand velocity during pole control and brain control with hand 
movements using 5 min epochs for training the model and testing predictions 


Brain control model 
trained in brain 


Brain control model 
trained in pole 


Pole control (a) control (b) control (c) 
Monkey 1 
Prediction of V, 0.63 + 0.02 b, c 0.39 = 0.01 a, ¢ 0.44 + 0.01 a, b 
Prediction of Vy, 0.65 + 0.02 b, c 0.42 + 0.01a 0.45 + 0.01a 
Monkey 2 
Prediction of V, 0.55 + 0.01b, c 0.40 = 0.01a 0.43 + 0.02 a 
Prediction of UF 0.40 + 0.01 b, c 0.20 + 0.01a 0.25 + 0.02a 


Numbers represent mean R (correlation coefficient between actual and predicted velocities) + SD and the columns 
for which Tukey's multiple comparison test indicated significant difference (p < 0.05). 


to maintain performance. Previously, we reported (Carmena et 
al., 2003) that, as a consequence of the increase in correlation 
between the neurons during BCWOH, PDs of many neurons 
became more similar (Fig. 6A). This effect may appear maladap- 
tive, because PD diversity is likely to improve directional encod- 
ing (Georgopoulos et al., 1986, 1988). However, our present anal- 
ysis using the shuffling procedure showed that tuning of 
individual neurons was increased because of the correlation be- 
tween neurons. Thus, the neuronal ensemble controlling the BMI 
may have been optimizing its performance by finding a trade-off 
between the magnitude of tuning and the diversity of PDs. It is 
possible that increased correlation was a temporary effect, and 
with prolonged and intensive training to control the BMI, the 
neuronal ensemble would eventually return to the original distri- 
bution of PDs and neuronal correlations. Even during learning of 
relatively simple motor skills, highly distributed cortical net- 
works undergo transient changes before settling to the level of 
automated performance, as shown by human neuroimaging 
studies (Jueptner et al., 1997; Floyer-Lea and Matthews, 2004). 
Interestingly, epochs of high attention to motor performance 
(and increased attention is very likely needed for successful learn- 
ing to control a BMI) have been found to be associated with 
increases of synchrony between neurons (Murthy and Fetz, 1992, 
1996a,b; Riehle et al., 1997, 2000; Hatsopoulos et al., 1998; Leb- 
edev and Wise, 2000; Baker et al., 2001). 


Off-line predictions of hand velocity under different conditions (key on top). A, Neuron-dropping curves depicting 
hand prediction quality as a function of neuronal sample size. B, Temporal dynamics of prediction quality for the entire ensemble. 
C Prediction quality estimated using a single 100 ms bin placed at different lags with respect to IVM. 
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In conclusion, we propose that a wide- 
spread process of cortical plasticity, result- 
ing from BMI operation, may lead to the 
establishment of a representation of an ar- 
tificial actuator into multiple cortical ar- 
eas. This raises the interesting hypothesis 
that the distinct levels of primate profi- 
ciency in tool manufacturing and utiliza- 
tion may be defined by the ability of these 
animals to incorporate the properties of 
these tools as extensions of the subject’s 
own body. 
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Brief Communication 


Stable Ensemble Performance with Single-Neuron 
Variability during Reaching Movements in Primates 
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‘Department of Neurobiology, ?Center for Neuroengineering, *Department of Biomedical Engineering, and ‘Department of Psychological and Brain 
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Significant variability in firing properties of individual neurons was observed while two monkeys, chronically implanted with multielec- 
trode arrays in frontal and parietal cortical areas, performed a continuous arm movement task. Although the degree of correlation 
between the firing of single neurons and movement parameters was nonstationary, stable predictions of arm movements could be 
obtained from the activity of neuronal ensembles. This result adds support to the idea that movement parameters are redundantly 
encoded in the motor cortex, such that brain networks can achieve the same behavioral goals through different patterns and relative 
contribution of individual neuron activity. This has important implications for neural prosthetics, suggesting that accurate operation of 
a brain-machine interface requires recording from large neuronal ensembles to minimize the effect of variability and ensuring stable 


performance over long periods of time. 


Key words: variability; neuronal ensemble; reaching; brain-machine interface; monkey; motor cortex 


Introduction 

Numerous studies have shown that cortical neurons modulate 
their firing in association with spatial and temporal aspects of 
behavior (Fetz, 1992; Johnson et al., 2001). In recent years, the 
idea that neuronal modulations can be used to control a brain— 
machine interface (BMI) has received significant attention 
(Wessberg et al., 2000; Nicolelis, 2001; Taylor et al., 2002; Car- 
mena et al., 2003; Nicolelis, 2003; Musallam et al., 2004; Lebedev 
et al., 2005). There are, however, different opinions in the litera- 
ture about the size of the neuronal sample needed to operate a 
BML efficiently. One principal challenge in building useful BMIs 
is coping with variability of neuronal firing (Zohary et al., 1994; 
Abbott and Dayan, 1999). Cortical neurons do not represent any 
motor parameter precisely (Ashe and Georgopoulos, 1994; Ser- 
gio and Kalaska, 1998; Carmena et al., 2003). Instead, neuronal 
firing fluctuates from one behavioral trial to another (Lee et al., 
1998; Shadlen and Newsome, 1998; Cohen and Nicolelis, 2004). 
This variability in neuronal firing and its correlation with behav- 
ioral parameters has recently been explained by a model of neural 
computation based on perturbations (Mass et al., 2002). This 
model suggests that stable internal states are not required for a 
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stable output in highly dimensional dynamic systems like the 
brain. 

Here, we report significant variability in firing properties of 
individual neurons while two monkeys, chronically implanted 
with multielectrode arrays in frontal and parietal cortical areas, 
performed a continuous arm movement task. The activity of a 
single neuron was typically correlated with several motor param- 
eters. However, the strength of a given correlation was generally 
nonstationary within a 30—60 min recording session. Moreover, 
although the correlation between the firing of single neurons and 
movement parameters was variable, stable predictions of arm 
movements could still be obtained from the activity of the whole 
neuronal ensemble recorded in each session. Thus, although sig- 
nificant variability was observed in the firing properties of indi- 
vidual neurons, the ensemble performance in the BMI control 
remained stable. 

This result adds to the recent literature that demonstrates re- 
dundant encoding by neuronal ensembles (Laubach et al. 2000; 
Schneidman et al., 2003; Cohen and Nicolelis, 2004; Paz and 
Vaadia, 2004; Narayanan et al., 2005) and shows that brain net- 
works can achieve the same behavioral goals through different 
patterns and relative contribution of individual neuron activity. 
The fact that the brain may use different ensembles of neurons to 
perform a given task may impact the design and implementation 
of a robust brain—machine interface that can maintain a stable 
performance over time. 


Materials and Methods 


Behavioral training and electrophysiology. Two adult female monkeys 
(Macaca mulatta) were used in this study. All procedures conformed to 
the Guide for the Care and Use of Laboratory Animals and were approved 
by the Duke University Institutional Animal Care and Use Committee. 
Before the surgical implantation of recording arrays, the animals com- 
pleted a period of chair training. The arrays contained 16—64 microwires 
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Figure1. Experimental design and examples of stable and variable neurons. A, Schematics of the behavioral task and neuronal 
encoding analysis. The monkey controls a handheld pole while seated in front of a computer screen (/). The goal of the task is to 
move the cursor (small disk) to a visual target (large disk) that appears at random locations on the screen for each trial. Multidi- 
mensional linear regression is applied off-line for linearly mapping neuronal ensemble activity with motor output (if). Indepen- 
dent models are generated for hand position and hand velocity coordinates. For each sample (bin size, 100 ms), 10 lags (i.e., 1s 
total) before movement are assigned as free parameters to the linear model. B, Examples of stable and variable neurons during a 
22 min recording session of continuous movement (first 10 min used for fitting the first model are not shown). Single-unit 
waveforms from PMd (top) and M1 (bottom) neurons recorded during the same session (/). Numbers in vertical axes depict the 
mean Vop (in microvolts) of the action potentials. Interspike intervals (ISIs) of the depicted neurons are shown (ii). Evolution of the 
IRFs obtained by regressing a single unit with each motor parameter are shown (iif). Models were generated with data points from 
a 10 min sliding window in 30 s increments. Each model consisted of 10 free parameters from the corresponding time lags. The 
color bar denotes IRF values for each model. Note the different IRF patterns across time between a stable (top) and variable 
(bottom) neuron. Evolution of the single-unit ranking, single-unit R score, and ensemble score across the recording session are 
shown (iv). For plotting purposes, neuron ranking is normalized from 0 (worst) to 1 (best) from the 94-neuron optimized 
ensemble. Notice the abrupt changes in both ranking and R score of the variable (bottom) neuron. 


each (separation between adjacent microwires, 300 zm) and were im- 


planted in frontal and parietal cortical areas. Implanted areas specifically subsequent records. 
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handheld pole. The position of the monkey’s 
hand was obtained from an infrared marker lo- 
cated on top of the pole. The marker was mon- 
itored by an optical tracking system (Optotrak; 
Northern Digital, Waterloo, Ontario, Canada). 
In the task, the monkeys were shown a small 
disk (the “cursor”) and a larger disk (the “tar- 
get”). This is illustrated in Figure 1 Ai. Each trial 
began with a target presented at a random po- 
sition on the screen. The monkey had to use the 
pole to put the cursor over the target for at least 
150 ms. If the monkey crossed the target too 
fast, the target disappeared and the trial was not 
rewarded. The monkeys had 5 s to hit the target, 
with a delay of 0.5 s in between trials. They re- 
ceived juice rewards for correct performance. 

Linear model. Hand position, velocity, and 
gripping force were modeled as a weighted lin- 
ear combination of neuronal activity using a 
multidimensional linear regression or Wiener 
filter (Haykin, 2002) described by the following 
equation: 


n 


y(t) =b+ > a(u)x(t — u) + e(t). 


u=—m 


(1) 


In this equation, x(t — u) is an input vector of 
neuronal firing rates preceding time t by time- 
lag u, y(t) is a vector of kinematic variables (e.g., 
hand position and velocity) at time ft, a(u) is a 
vector of weights for time-lag u, b is a vector of 
y-intercepts, and e(t) are the residual errors. 
The lags can be positive (i.e., past) or negative 
(future). Here, we considered only lags in the 
past. This equation can be recast in matrix form 
as follows: 


Y=XA, (2) 


where each row in each matrix corresponds to a 
unit of time and each column is a data vector. 
Matrix X contains lagged data, and thus has a 
column for each lag multiplied by each neural 
channel (e.g., 100 channels by 10 lags to a total 
of 1000 columns). The y-intercept is incorpo- 
rated by prepending a column of ones to matrix 
X. Matrix A is solved by the following equation: 


(3) 


Neuronal firing rates were sampled using 100 
ms bins. Ten bins preceding time t were used for 
training the model and consequent predictions. 
Thus, in Equation 1, m and n equal 0 and 9, 


A = inv(X'X)X'Y. 


respectively. Models were trained with 10 min of data and tested on the 


included the dorsal premotor cortex (PMd), supplementary motor area 
(SMA), and the primary motor cortex (M1) in both hemispheres. In 
monkey 1, an additional implant was placed in the primary somatosen- 
sory cortex (S1). In monkey 2, the medial intraparietal area (MIP) of 
posterior parietal cortex (PP) was also implanted. Both monkeys per- 
formed the task with their left arms. After recovery from the surgical 
procedure, the animals were engaged in behavioral training. Neuronal 
recordings commenced at the same time. A 512 Multichannel Acquisi- 
tion Processor (Plexon, Dallas, TX) simultaneously recorded single- 
neuron and multiunit activity during each recording session. Spike sort- 
ing was performed as described previously (Nicolelis et al., 2003). In this 
study, only single units that maintained stable waveforms across the 
analyzed recording sessions were used. Monkeys were seated in a primate 
chair facing a computer monitor. They performed a motor task using a 


Data analysis and variability measurements. Data from 10 recording 
sessions for each monkey were used in this study. For each session, a 
linear model was generated for each of the motor parameters investigated 
in this study (ie., X and Y coordinates of hand position and velocity) (Fig. 
1 Aii). Models were fitted using training data sets of 10 min with a 30 s 
step sliding window. All of the models generated in a given session were 
tested against a data set of 5 min drawn from the end of the session. For 
assessing neuronal variability, firing rates of individual neurons were 
regressed independently and subsequently tested for prediction of each 
motor parameter. As with ensemble regression, regressions for individ- 
ual neurons used 10 time lags, resulting in 10 free parameters as inputs to 
the model. Dividing by the SD before conducting linear regression nor- 
malized firing rates of each neuron. Prediction quality was evaluated as 


302 


10714 « J. Neurosci., November 16, 2005 - 25(46):10712-10716 


the correlation coefficient R between the 5 min test data set and the 
predictions from the model. The values of R were then ranked from the 
highest (first rank) to the lowest. In addition to R-based ranking, neuro- 
nal sensitivity was used to describe the contribution of neurons to en- 
semble predictions. Neuronal sensitivity was calculated as the average of 
the absolute value of the impulse response functions (IRFs) obtained 
from the regression for the whole ensemble (Sanchez et al., 2004). Both 
methods produced similar results (r = 0.88). Significant variability in the 
contribution of individual neurons to prediction of motor parameters 
was detected as the presence of fluctuations in ranking across a recording 
session that exceeded 3o. If the contribution rank of neuron i at time tf 
exceeded 30 from the mean of the distribution of the remaining ranking 
points in that session, the neuron was classified as variable. The fact that 
this selection criterion is not sensitive to smooth ranking transitions 
across the session, but rather is only affected by abrupt decreases or 
increases in rank value, makes it very conservative. The same criterion 
was applied to measure variability in the mean firing rate of individual 
neurons. 

Cumulative and residual neuron ranked curves. Ten recording sessions 
were used for this analysis. Prediction capacity of each neuron in the 
ensemble during each session was ranked as described above. For the 
cumulative ranked curves, models were trained (10 min epochs) and 
tested (5 min epochs) for each motor parameter. The size of the popula- 
tion was increased in unitary increments, starting from one (best) single 
unit and ending with the total size of the ensemble. For the residual 
ranked curves, the procedure was the inverse (i.e., starting with the whole 
ensemble and removing one neuron at a time starting with the neurons 
with the highest ranks and ending with the neurons with the lowest 
ranks). The final cumulative and residual curves represented the average 
for all sessions. 


Results 


In this study, we examined the relationship between neuronal 
activity and hand movements. We used 94 simultaneously re- 
corded units in monkey 1 and 38 in monkey 2, all of which had 
stable action potentials within all recording sessions. Neuronal 
variability was quantified in terms of significant fluctuations 
( p < 0.01) of correlation between neuronal activity and param- 
eters of hand movements. In the neuronal ranking analysis, this 
correlation was compared for different neurons in the ensemble. 

We first examined firing patterns of individual neurons re- 
corded from different cortical areas during several days of the 
monkeys’ performance in the task. Individual neurons were clas- 
sified in two main groups depending on the degree of variability 
in their encoding characteristics in each recording session. If the 
ranking of a neuron fluctuated significantly (p < 0.01) during 
the session, the neuron was classified as variable; otherwise the 
neuron was classified as stable. Figure 1B shows representative 
neurons from each group recorded in the same recording session 
in monkey 1. The evolution of the properties of each neuron 
during the course of the session is represented using the IRFs 
from the linear model obtained by regressing neuronal activation 
with hand position ( y-coordinate) (Fig. 1 Biii). Whereas the IRFs 
of the stable neuron remained very similar across the session, the 
IRFs of the variable neuron changed dramatically. The result of 
testing the models (i.e., the sum of products between IRFs and the 
spike trains of their corresponding neurons) is shown in Figure 
1 Biv. For the stable neuron, none of the curves representing the 
ranking and contribution of the neuron (R score) changed signif- 
icantly over time. The variable neuron (Fig. 1 Biv, bottom), how- 
ever, showed dramatic changes both in contribution to the model 
(blue line) and in ranking (green line) during the first 3 min of the 
session, whereas the ensemble encoding remained stable. Notice 
the large fluctuations in ranking of the neuron; it was the worst 
performer in the beginning of the session and became the best by 
the end of the session. 
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Next, we examined the spatiotemporal patterns of single- 
neuron variability based on ranking changes across the whole 
ensemble in monkey 1 (Fig. 2A). Individual neuronal activity was 
regressed to hand velocity (X component) in a recording session 
from monkey 1. Figure 2 Ai shows neuronal variability in ranking 
(1, best; 94, worst) across all cortical areas, including M1, which 
was the best area for encoding movements. Sorting the ensemble 
by ranking in the first epoch (Fig. 2 Ai7) and plotting significant 
variable epochs ( p < 0.01) for each neuron (Fig. 2 Aiii), shows 
that only the best four neurons of the ensemble remained rela- 
tively stable across the whole session. Up to 10% of the total 
population could simultaneously exhibit significant variability in 
a given epoch without affecting ensemble predictions (Fig. 2 Aii, 
top mono-dimensional plot). For a given neuron, up to 16% of 
the epochs within a recording session were classified as signifi- 
cantly variable (Fig. 2 Aii, right mono-dimensional plot). Neu- 
rons that provided a high contribution to a parameter in a given 
epoch were not necessarily the best contributors for predictions 
of the same motor parameter during the other epochs. This effect 
was observed for all cortical areas. Variability in a given session 
was quantified as a percentage of neurons of the ensemble that 
showed changes of >3 SDs at least once during the session. Figure 
2B shows the variability of the rankings that resulted from pre- 
dictions of all motor parameters and the variability of the mean 
firing rate of the individual neurons across four consecutive ses- 
sions. In monkey 1, variable neurons reached 65% of the ensem- 
ble size for ranking and 55% for firing rate. In monkey 2 (data not 
shown), these values reached 54 and 50%, respectively. Although 
individual neurons exhibited dramatic changes, overall ensemble 
performance did not decrease significantly. Ensemble perfor- 
mance was stable because of the compensation of other neurons 
contributing to the linear-model predictions. The compensation 
is revealed in Figure 2C, which shows redundancy of information 
in the ensemble. In the figure, cumulative curves show that the 
optimal linear mapping between spike trains and motor output 
can be obtained using the top 50 neurons and, more remarkably, 
that ~90% of the accounted variance is obtained with only the 
first top 10 neurons. In residual curves (i.e., the performance of 
the remaining ensemble after individually removing neurons in 
ranked order), the figure shows the rest of the ensemble contain- 
ing a considerable amount of useful information. For example, 
after removing the best 50 neurons, 60% of the total ensemble 
contribution remained in the residual population, suggesting 
that the majority of the remaining ensemble was not composed of 
noisy, task-unrelated neurons but rather of highly modulated 
task-related neurons. 

Finally, Figure 2D illustrates the importance of recording 
from large neuronal ensembles to maximize and stabilize model 
performance. The figure compares performance, for each epoch, 
between ensembles of 10 randomly selected neurons (mean R + 
SD; 15 samples) and ensembles of the top 10 neurons of a total 
population of 94 units. The large size of this population increases 
the chance of yielding neurons with large contributions for a 
given epoch, resulting in significantly higher and more stable 
performance than with small populations as represented by the 
randomly chosen ensembles. 


Discussion 

The analysis performed in this paper was motivated by the obser- 
vation that the neurons that provided the highest contribution to 
a given parameter during a given epoch did not necessarily pro- 
vide the highest contribution to the same parameter during other 
epochs. Additional analysis revealed that, although there was sig- 
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tions of redundancy in neuronal ensembles. A, Spatiotemporal patterns of single-neuron variability based on ranking changes 
during a recording session. A color map shows an ensemble of neurons arranged by cortical area (i). The color bar denotes ranking 
(1, best; 94, worst) and the range of individual correlation coefficient scores for the whole ensemble. Arrows depict the top four 
neurons. A color map shows the same ensemble of neurons as in i but arranged by their initial ranking at the first epoch (if). The 
mono-dimensional plot on top shows stable ensemble performance. A color map (red color) depicts statistically significant 
variability ( p < 0.01) ofa particular neuron at a particular epoch (iif). The ensemble is arranged as in ii. Mono-dimensional plots 
show the amount of significantly variable neurons at a given epoch (top) and the amount of epochs for a given neuron that were 
classified as variable within the recording session (right). B, Overall quantification of neuronal variability across all motor param- 
eters and for four consecutive recording sessions. Variability was quantified as statistically significant ( p < 0.01) changes in 
neuron ranking and firing rate. The numbers in parentheses indicate the size of the neuronal population in a particular cortical 
area. C, Cumulative (solid) and residual (dotted) neuron-ranked curves for position (X, Y) and velocity (VX, VY). Notice the smooth 
decay of the residual curves showing the high contribution of most of the neurons of the ensemble. D, Performance comparison 
between ensembles of the best 10 neurons in a given epoch and ensembles of 10 randomly selected neurons, of a total population 
of n = 94 units. For each epoch, 15 ensembles of randomly selected neurons were estimated (mean R = STD). 
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neuronal variability is simply biological 
noise. The stable activity of the ensemble 
might then arise because the noise contrib- 
uted by more variable neurons remains 
approximately constant across the ensem- 
ble, although the contribution of individ- 
ual neurons changes stochastically. Al- 
though the underlying basis of this 
neuronal variability is unknown, one pos- 
sibility is that the neuronal population 
may have a bias for encoding other poten- 
tial variations of movements that are not 
exercised in the current paradigm. In this 
regard, additional experiments with be- 
havioral tasks containing sequential 
movements in different contexts (Ben- 
Shaul et al., 2004) will be needed to further 
investigate this phenomenon. 

It is tempting to speculate that neuro- 
nal variability is beneficial for neuronal 
computations (for example, fault toler- 
ance in a highly redundant system like the 
brain). In fact, the redundancy of encod- 
ing by neuronal ensembles that we ob- 
served concurs with recent reports 
(Schneidman et al., 2003; Ben-Shaul et al., 
2004; Cohen and Nicolelis, 2004; Naray- 
anan et al., 2005). As shown in Figure 2C, 
the redundant information that remains 
after removal of the best performing neu- 
rons is depicted by the smooth decay of the 
residual curves. This redundant informa- 
tion allows the ensemble to maintain opti- 
mal performance across time and supports 
our previous contention (Wessberg at al., 
2000; Carmena et al., 2003) that reliable 
task performance in BMIs will require re- 
cording from large neuronal ensembles. 
Specifically, by sampling over a large pop- 
ulation of neurons, the effect of individual 
neuronal variability on the ensemble out- 
put will be minimized. Moreover, by re- 
cording from many neurons, the BMI will 
be able to select a subpopulation of neu- 
rons with the greatest contribution to 
achieve optimal task performance in the 
control of a prosthetic device at a given 
moment. Consistent with the results re- 
ported here, Kurtzer et al. (2005) reported 
random shifts in load representation dur- 
ing behavioral tasks that involved control 
of posture and reaching movement. The 
authors argue that current BMI ap- 
proaches that assume fixed correlation be- 
tween neuronal activity and motor output 
may be suboptimal. Therefore, by record- 
ing from large neuronal ensembles, it will 
be possible to overcome variable represen- 
tations by fitting separate models for spe- 


nificant variability in neuronal properties, the performance ofthe cific components of the behavioral task. It should also be noted 
whole ensemble was stable. One explanation for the ensemble _ that recording from many neurons also protects the performance 
stability is that the change in performance of one neuron was _ of the BMI from the loss of neuronal activity to electrode mal- 
compensated by other neurons. Another explanation is that the — function, modification of electrode tip properties, or cell death. 
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BMIs that rely on only a very small sample of neurons (Serruya et 
al., 2002; Taylor et al., 2002) are not likely to accommodate the 
alterations in the neuronal sample over time. 

In conclusion, this study showed that, although individual 
cortical neurons can exhibit significant variability in both firing 
properties and correlation with a motor task within an ensemble, 
the ensemble can achieve the same behavioral goals through dif- 
ferent patterns and the relative contribution of the activity of 
individual neurons. This finding reinforces the importance of 
recording from large neuronal ensembles to achieve reliable BMI 
performance in the presence of individual neuronal variability. 
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Both humans and animals can discriminate signals delivered to sensory areas of their brains using electrical microstimulation. This 
opens the possibility of creating an artificial sensory channel that could be implemented in neuroprosthetic devices. Although micro- 
stimulation delivered through multiple implanted electrodes could be beneficial for this purpose, appropriate microstimulation proto- 
cols have not been developed. Here, we report a series of experiments in which owl monkeys performed reaching movements guided by 
spatiotemporal patterns of cortical microstimulation delivered to primary somatosensory cortex through chronically implanted multi- 
electrode arrays. The monkeys learned to discriminate microstimulation patterns, and their ability to learn new patterns and new 
behavioral rules improved during several months of testing. Significantly, information was conveyed to the brain through the interplay 
of microstimulation patterns delivered to multiple electrodes and the temporal order in which these electrodes were stimulated. This 
suggests multichannel microstimulation as a viable means of sensorizing neural prostheses. 


Key words: microstimulation; brain-machine interface; primate; somatosensory; discrimination; neuroprosthetics 


Introduction 

For more than a century, electrical stimulation has been used to 
probe brain circuitry and function in both humans (Penfield and 
Boldrey, 1937; Penfield and Rasmussen, 1950; Ransom, 1892) 
and animals (Fritsch and Hitzig, 1870; Ferrier, 1873, 1875). The 
effects of electrical stimulation depend on the location and pa- 
rameters of stimulation. Stimulation of large brain areas typically 
hinders information processing (Pascual-Leone et al., 2000; 
Chambers and Mattingley, 2005; Kammer, 2006). Conversely, 
stimulation of small areas, termed microstimulation, can evoke 
motor and sensory effects that mimic the functional contribution 
of the stimulated area (Tehovnik, 1996; Graziano et al., 2002; 
Cohen and Newsome, 2004; DeAngelis and Newsome, 2004; 
Tehovnik et al., 2006). In the 1950s, neurobiologists began 
operantly conditioning animal behavior using electrical stimula- 
tion of the brain as a conditioned stimulus (Doty et al., 1956; 
Nielson et al., 1962; Doty, 1965, 1969) or a reinforcement (Olds 
and Milner, 1954). More recently, microstimulation of sensory 
areas has been shown to produce perceptual effects such as visual 
sensation (Bartlett et al., 2005), biasing the perception of visual 
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motion (Salzman et al., 1990, 1992; Britten and van Wezel, 1998) 
or face recognition (Afraz et al., 2006), and mimicking the so- 
matosensory perception of flutter (Romo et al., 1998, 2000; de 
Lafuente and Romo, 2005). Impressively, Talwar et al. (2002) 
used microstimulation of the rat barrel cortex to guide rats 
through a complex terrain. Although it is often difficult to prove 
that animals experience perceptions during microstimulation, 
microstimulation-induced perceptions have been demonstrated 
in humans (Penfield and Perot, 1963; Brindley and Lewin, 1968; 
Dobelle et al., 1976; Davis et al., 1998; Kiss et al., 2003; Ohara et 
al., 2004). 

With the development of multielectrode implants (Nicolelis 
et al., 2003) and the concurrent advances of brain—machine in- 
terfaces (Chapin et al., 1999; Wessberg et al., 2000; Taylor et al., 
2002; Carmena et al., 2003; Lebedev et al., 2005; Lebedev and 
Nicolelis, 2006), there is renewed interest in microstimulation as 
a means of providing the brain with an artificial sensory channel. 
Such a channel could recover sensation lost because of a neuro- 
logical disorder or it could convey information from sensors of a 
prosthetic limb (Berger et al., 2005; Middlebrooks et al., 2005; 
Lebedev and Nicolelis, 2006; Wickelgren, 2006). Although this 
idea is intriguing, two critical issues must be addressed: (1) 
whether such artificial sensation can be improved by using mul- 
tichannel microstimulation, and (2) whether microstimulation 
in this application is suitable for long-term usage. We explored 
the first issue by testing the capacity of owl monkeys to discrim- 
inate multichannel microstimulation of increasing complexity. 
We investigated the second issue by testing the long-term efficacy 
of microstimulation. 

Two monkeys were previously trained in a reaching task in 
which target location was cued by vibrotactile stimuli (Sandler, 
2005). Here, skin vibration was replaced by microstimulation of 
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Food Reward hidden 
\\..| behind one of two doors 


Figure 1. Experimental setup. Owl monkeys, while behind a transparent barrier, were cued 
to the location of a food reward through cortical microstimulation delivered to $1 in the left 
hemisphere. After a 1 s delay following microstimulation, the barrier was raised, and the mon- 
keys reached with the right arm to one of two doors. After opening the correct door, they 
retrieved a food reward, or alternatively after opening the incorrect door, they did not get any 
reward. 


the primary somatosensory cortex (S1) delivered through multi- 
ple electrodes. During these long-term experiments, the mon- 
keys’ performance and their capability to learn new behavioral 
contingencies steadily improved. 


Materials and Methods 


Two adult female owl monkeys (Aotus trivirgatus) were implanted pre- 
viously (Sandler, 2005) with microwire arrays in several cortical areas: 
primary somatosensory (S1), primary motor (M1), dorsal premotor, and 
posterior parietal cortices. These implants remained viable for electro- 
physiological recordings after more than four years. In the experiments 
reported here, we used S1 implants to deliver microstimulation patterns 
of different levels of complexity. S1 sites used for microstimulation cor- 
responded to cutaneous receptive fields located on the monkeys’ hands. 
In monkey 1, two of the stimulation sites had clear receptive fields in the 
central palm of the right hand, whereas the remaining two sites corre- 
sponded to glabrous skin of the index and middle fingers of the same 
hand. In monkey 2, receptive fields of the stimulated sites were on the 
hand, although not as clearly defined as in monkey 1. These receptive 
fields were different from the location of the vibratory cues, which were 
delivered to the shoulder. Stimulation parameters (current amplitude of 
100-150 pA, pulse duration of 0.1 ms anda rate of pulse trains of 100 Hz) 
were in the same range as that used previously for S1 microstimulation in 
rhesus monkeys (Romo et al., 1998, 2000; de Lafuente and Romo, 2005), 
as well as in other cortical microstimulation studies (Murasugi et al., 
1993; Tehovnik et al., 2006). 

Here, multichannel cortical microstimulation was used as a cue that 
guided reaching movements. Each monkey performed a reaching and 
grasping task that consisted of retrieving a food item from one of two 
locations covered by opaque doors (Fig. 1). Both monkeys were trained 
previously to guide their reaches by peripheral vibrotactile cues applied 
to their upper arms (Sandler, 2005). In this study, we used cortical mi- 
crostimulation to cue the same behavior. Additionally, the initial time 
course (Fig. 2 B) of the microstimulation pulses was the same as the time 
course for the vibratory stimulus that had been learned previously, ex- 
cept the microstimulation was delivered at 100 Hz instead of the 60 Hz 
used in vibratory stimulation. 

The complexity of microstimulation patterns was progressively in- 
creased as the monkeys learned new discrimination tasks (Fig. 2). The 
patterns delivered to each electrode consisted of 100 Hz trains of electri- 
cal current pulses that were alternatively turned on and off, forming 
temporal patterns. Spatial patterns of microstimulation were formed by 
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triggering microstimulation on different electrodes in different orders: 
“waves” of microstimulation were created, which propagated in different 
directions (Fig. 2 E). Importantly, in all cases in which the monkeys dis- 
criminated between two microstimulation patterns, these patterns were 
never different in the total number of electrical pulses delivered through 
the stimulating electrodes. The difference was in the temporal and/or 
spatial patterns of microstimulation, not in the electrical charge passed 
through the electrodes. 

Surgical and electrophysiological procedures. All surgical and behavioral 
procedures conformed to the National Research Council Guide for the 
Care and Use of Laboratory Animals and were approved by the Institu- 
tional Animal Care and Use Committee. During the implantation sur- 
gery (Kralik et al., 2001), each of the two adult female owl monkeys (A. 
trivirgatus), weighing 800-850 g, was anesthetized using 1.5—2.0% isof- 
lourene and placed in a stereotaxic apparatus. A series of small craniot- 
omies was made, both to grant access to the brain for the microwire 
arrays and for anchoring the dental acrylic to the skull. In each animal, 
multiple microwire arrays were chronically implanted in several cortical 
areas. The cortical areas implanted were primary somatosensory (S1), 
primary motor (M1), dorsal premotor (PMd), and posterior parietal 
(PP) cortices. Either 2 X 8 or 4 X 8 arrays of 50 wm Teflon-coated steel, 
spaced 300 yum apart, were used. The placement of the electrodes was 
accomplished using stereotaxic coordinates. Connectors for the arrays 
were embedded in a head cap made of dental acrylic. 

Electrophysiological recordings and microstimulation. During the exper- 
imental sessions, the monkeys were placed in a custom-designed chair 
enclosure that restricted them to a sitting position while allowing for free 
arm motion and minimal head motion. Head stage amplifiers, or a cus- 
tom microstimulation adapter were attached to the head-cap connectors. 
A flexible wire harness, in turn, connected the headstages to a 96-channel 
Plexon (Dallas, TX) recording system and the custom microstimulation 
adapter to the microstimulation device. The Plexon system was used for 
receptive field testing and, in concluding sessions, for obtaining neuronal 
recordings simultaneously with microstimulation. 

We used several patterns of microstimulation. Initially, microstimu- 
lation was delivered to S1 through pairs of Iso-Flex stimulus isolators 
operating in constant current mode, with one isolator per pair producing 
the anodic current pulse and the other producing the cathodic current 
pulse. Triggering pulses were delivered to the isolators via the A.M.P.I. 
Master-8 programmable stimulator. The isolators were interfaced with 
the connectors mounted in the head caps via custom cabling. Later, this 
microstimulation hardware was replaced with a custom built, four- 
channel, constant-current, biphasic, bipolar microstimulator controlled 
by custom software. The output voltage of each channel was supplied by 
two cascaded 1 W miniature isolated switching power supplies; these, in 
series, generated voltages up to 90 V. This voltage was switched to the two 
bipolar electrodes through an optically isolated h-bridge in series with a 
simple current control circuit. The latter consisted ofan optically isolated 
digital to analog converter (DAC), a current sense resistor, an opamp, 
and MOSFET (metal-oxide semiconductor field-effect) transistor in a 
feedback topology where the regulated current was set by the DAC out- 
put. Input to the DAC, like the h-bridge, was optically isolated; to con- 
serve digital lines, the anodic and cathodic enable lines were switched via 
a global signal to control serial input to the 12-bit DAC register. The 
digital control lines for enabling the anodic and cathodic legs of the 
h-bridge were controlled directly by a National Instruments PCI-6533 
card. 

This stimulator was run continuously at a sampling rate of 100 kHz via 
card-initiated direct memory access (DMA) through customized Linux 
device drivers. The DMA buffer was, in turn, serviced bya C+ + program 
with, effectively, up to four nested periods and duty cycles. These param- 
eters were controlled via a Python script. The script permitted the user to 
control the stimulator from the Internet or programmatically via http 
GET commands. The Python script supported arbitrary loadable presets, 
which were stored in a MySQL database on the same Linux computer. 

Stimulation parameters were similar to those used previously in S1 
microstimulation in rhesus monkeys (Romo et al., 1998, 2000). These 
parameters were first tested by stimulating the motor cortex. We started 
with very low amplitude stimulation via electrodes in motor cortex, grad- 
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designate the sequential stimulus and ground electrode pairs along the linear electrode array. 


ually raising the amplitude and frequency until a motor response was 
detected. Based on these estimations, we chose an initial stimulating 
current amplitude of 100-150 mA, pulse duration of 0.1 ms, with an 
interpulse interval of 0.05—0.1 ms to avoid any interaction between the 
pulses because of imperfect stimulus decays of the two pulses making up 
the biphasic, and a rate of pulse trains of 100 Hz. The pulse duration used 
in this study was the result of our empirical exploration to find the 
shortest pulse duration that worked, thus helping to avoid long-term 
tissue damage. Additionally, there were two issues that prevented the 
delivered current pulses from being ideal square pulses, both of which we 
compensated for. First, the components of the stimulation circuit (elec- 
trodes and custom wiring harness) added a capacitive load to the stimu- 
lator. Second, the optical components used the stimulation hardware had 
limited switching speed. In our setup, the switching limitations had a 
slightly larger effect than the capacitive terms, and the combination of 
these two created approximately exponential ramps, which reached 90% 
of their maximum in ~30 ps. To compensate for the decrease of the 
delivered charge that resulted from these ramps, the duration of the 
command pulse was lengthened, making the net charge delivered equal 
to that which would have been delivered by a perfect rectangular pulse. 
The injected charge, which induces an electric field, is the parameter 
most related to the efficacy of stimulation. Because charge is a product of 
pulse width (or duration) and amplitude, both of these parameters in- 
fluence the efficacy of microstimulation. In our analyses of psychometric 
curves, we modulated amplitude to modulate charge delivery, while 
keeping pulse width fixed. 

Behavioral task. Two owl monkeys performed a reaching and grasping 
task. Each monkey sat comfortably in a primate chair. Two Plexiglas 
screens, one opaque, one transparent, separated the monkey from two 
small opaque doors, one to the right of the midline and one to the left. 
Food was hidden behind one of the doors, and the monkeys were behav- 
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Temporal patterns of microstimulation. A, In all sessions, stimulation pulses were delivered biphasically, with a 
cathodic phase preceding an anodic phase of equal amplitude. Pulse width, pulse delay, and frequency were kept constant at 0.1 
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also used in the temporal discrimination task (D). By keeping the low-level stimulation parameters constant between the two cues 
in the temporal and spatiotemporal tasks, the absolute charge injection was kept constant. In E, electrode pairs (EP1—EP4) 
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iorally conditioned to choose the location of the 
food by either peripheral vibrotactile cues or 
intracortical microstimulation cues (Fig. 1). Af- 
ter the food reward was hidden, the opaque 
screen was lifted and trials began with a5 s pe- 
riod, during which the animal was behind the 
transparent Plexiglas screen and able to see the 
closed doors but unable to open them. During 
the first 4 s, vibrotactile or microstimulation 
cues were delivered. Then, after a 1 s pause, the 
transparent screen was raised, and the monkey 
reached toward and opened one of the two 
doors. If the door selection was correct, the 
monkey obtained the food located behind the 
door. The monkey could make only one selec- 
tion per reach, because the experimenter 
quickly took the food away ifthe monkey failed. 

Trials occurred once every 30 s on average 
with ~2 s variability in the intertrial delay. In- 
correct reaches were followed by repetition of 
the same type of trial. This design of trial pre- 
sentation prevented the animals from becom- 
ing biased to a particular door but also made the 
trials that followed incorrect trials special, because 
the monkeys could use the previous failure as a 
cue of food location. These special correction tri- 
als were treated separately in the analyses of the 
behavioral data. Successful trials were followed by 
trials where food location was determined using a 
random number generator. The monkeys per- 
formed 100 trials during each daily session. 

Each monkey was first overtrained to per- 
form the reaching task, which was cued by vi- 
brotactile stimuli applied to its upper arms 
(Sandler, 2005). If the right arm was vibrated, 
the animal had to reach toward the rightmost 
target, and if the left arm was vibrated, the cor- 
rect direction of reach was to the left. Later, ina 
reversal phase of the experiment, each monkey 
also learned to reach to the target opposite to the stimulated arm. After 2 
years of testing using arm vibration as the cue, the stimulus was switched 
to intracortical microstimulation. All other aspects of the task remained 
the same after the switch to microstimulation. Because the microstimu- 
lation would be delivered unilaterally, to the left primary somatosensory 
cortex only, in the months immediately preceding the switch from vibro- 
stimulation to microstimulation, both monkeys were overtrained on 
unilateral vibrostimulation discrimination, reaching performance levels 
above 85% correct. The monkeys had to detect the presence or absence of 
vibration on the arm contralateral to the site that would be stimulated 
when the vibrostimulation was replaced with electrical microstimula- 
tion. In this way, the necessary relearning required of the stimulation cue 
switch was minimized. 

Here, we report only the data obtained in the experiments in which 
microstimulation was used. The time course of the delivered pattern of 
microstimulation reproduced the previously used pattern of vibratory 
cues (Sandler, 2005). Microstimulation cues consisted of 100 Hz trains of 
electrical pulses that were alternatively turned on for 150 ms and turned 
off for 100 ms, this pattern being repeated 16 times (i.e., for 4s). Micro- 
stimulation amplitude was set to 100 A for monkey 1 and to 150 wA for 
monkey 2, because these values corresponded to levels slightly below the 
average threshold for eliciting motor responses using microstimulation 
of M1. Electrical pulses were delivered simultaneously to three S1 elec- 
trodes in a linear arrangement with an interelectrode separation of 300 
pum (Fig. 2). The stimulation electrodes shared a common global ground. 
In most sessions, the ground electrode located on the stimulation array 
away from the stimulation sites was connected to the stimulator ground. 
Alternatively, in some early sessions, the ground screws embedded in the 
dental acrylic implant were used. 
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In the basic task (Fig. 2B), the monkeys dis- 
criminated the presence or absence of micro- 
stimulation. The correct response was to reach 
toward the target contralateral to the cortical 
site being stimulated (i.e., $1 in the left hemi- 
sphere) if the stimulation was turned on and to 
reach toward the other target if there was no 
stimulation. Electrical pulses were delivered si- 
multaneously to three S1 electrodes in a linear 
arrangement with an interelectrode separation 
of 300 wm (Fig. 3). These microstimulation 
patterns closely resembled vibrotactile patterns 
in which the monkeys were previously over- 
trained (Sandler, 2005): 100 Hz trains of electri- 
cal pulses were alternatively turned on for 150 
ms and turned off for 100 ms, this pattern being 
repeated 16 times (Fig. 2A). After the monkeys 
learned the basic task, this contingency was re- 
versed: the monkey was required to reach to- 
ward the target ipsilateral to the site of stimula- 
tion (Fig. 2C). 

Additionally, to study the psychometric rela- 
tionship between amplitude and discrimina- 
tion in the basic task, five different stimulus am- 
plitudes were used, ranging from 0 to 200 wA in 
logarithmic steps. For a given session, micro- 
stimulation amplitude was fixed at a randomly 
selected value without replacement. 

After learning this task, the monkeys were 
required to discriminate between two temporal 
patterns of microstimulation (Fig. 2D). The 
amplitude, frequency, pulse width, and the 
number of pulses were held constant between 
the two different stimulation patterns. The dif- 
ference was that one pattern was made of 150- 
ms-long trains of pulses (pulse rate, 100 Hz) 
separated by pauses of 100 ms (i.e., the same 
pattern as in the basic task), whereas the other was made of 300-ms-long 
trains separated by pauses of 200 ms. The duration of each pattern of 
microstimulation was 4s. The first stimulation pattern previously used in 
the basic task (Fig. 2B) instructed the monkey to select the door con- 
tralateral to the stimulation site (the left hemisphere). The second pat- 
tern (Fig. 2D) instructed the monkey to select the door ipsilateral to the 
stimulation site. Stimulation was always delivered to S1 in the left 
hemisphere. 

In the spatiotemporal task, the monkeys were required to discriminate 
between two spatiotemporal patterns of microstimulation. Again, the 
amplitude, frequency, pulse width, and number of pulses were held con- 
stant with pulse trains lasting for 150 ms, separated by pauses of 100 ms. 
However, whereas in previous tasks, all electrodes delivered the same 
pattern simultaneously; in this case, the electrodes were broken down 
into four electrode and ground pairs (EP1—EP4). To instruct the monkey 
to select the contralateral door, EP1 was stimulated for 150 ms, then a 100 
ms pause, then EP2 for 150 ms, and so on, producing a wave of stimulation 
from EP1 to EP4. The four electrode-pair sequence (EP1 to EP4) was re- 
peated four times for a total of 4 s. To instruct the ipsilateral door choice, the 
direction of the microstimulation wave was reversed, going from EP4 to EP1. 
In both stimulation patterns, each individual electrode pair delivered the 
same current pulses, the only difference being the spatiotemporal order in 
which these electrode pairs were stimulated (Fig. 2 E). 

Analyses of behavioral data. Experimental trials during each session 
were split into two separate groups for analysis: random trials and cor- 
rection trials. Correction trials (see above) were treated separately be- 
cause the food location in these trials did not change from the previous 
trial. Thus, to obtain food in a correction trial, a monkey had to simply 
switch its direction of reach from the previous trial. These trials were used 
to prevent the monkeys from developing any biased strategies. Specifi- 
cally, the monkeys had previously selected the same door repeatedly, getting 
a food reward half the time without paying attention to the cue. The correc- 
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Implantation sites and stimulation electrode arrangement. In each monkey, implanted electrodes consisted of three 
2 X 8 electrode arrays placed in PMd, M1, and S1 cortices as well as a fourth 4 X 8 array placed in posterior parietal (PP) cortex. 
Only the S1 array (red) was used for microstimulation. Electrodes were 50 pzm Teflon-coated steel, spaced 300 jzm apart. A, Inthe 
basic, reversal, and temporal tasks, biphasic stimulation pulses were delivered to successive electrodes in the array implanted in 
$1 with a common global ground. To deliver instructions to the brain, two in the temporal discrimination task or three in the basic 
and reversal tasks electrodes were simultaneously stimulated. B, In the spatiotemporal task, four different electrode-ground pairs 
were stimulated sequentially. 


tion trials were analyzed separately, and this analysis confirmed that the 
monkeys indeed learned to switch the direction of reach. All other trials, 
called random trials, were characterized by food placement according to a 
computer-generated random sequence. These trials were analyzed to deter- 
mine cue discrimination performance, because the only information regard- 
ing food location available to the monkeys came from the cue. 

Discrimination performance was quantified as the proportion of ran- 
dom trials performed correctly (0-1 range, with 0.5 corresponding to a 
chance level of performance). The performance over a series of sessions 
was fit to a linear regression model: 


y= MX + Vorfser> (1) 


where y is discrimination performance, x is experimental session num- 
ber, m is the learning slope, and y,¢,.¢ is the initial performance. In 
addition, a sigmoidal model was used: 


Y scale 
Y= Tp Fam sieen) 1 Yosser (2) 


where Yoerser describes the minimum of the sigmoid, y,..). is the height of 
the sigmoid, x,,,esh is the threshold or inflection point of the sigmoid, and 
k is a factor describing the slope of the sigmoid (larger k values denote 
steeper slopes). For each fit, regression statistics were calculated to deter- 
mine the goodness of the fit. 

In addition to describing changes in performance in time, Equation 2 
was used for constructing psychometric curves in which x corresponded 
to microstimulation amplitude. 


Results 

Control sessions and correction trials 

In control sessions, we showed that the monkeys did not use any 
strategy related to the reward schedule, experimenter interaction, 
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or any other unseen source of information, to improve their per- 
formance. In these sessions, food was randomly placed exactly as 
in the basic task, but no cue was delivered to indicate food loca- 
tion. Mean performance on these random control trials was 0.48 
and 0.47 in monkeys 1 and 2, respectively, which was not signif- 
icantly different from chance (tf test; p > 0.05). 

Throughout all sessions, mean performance in correction tri- 
als, when food location was not altered following an incorrect 
trial, was 0.93 and 0.87 for monkeys 1 and 2, respectively. Perfor- 
mance in these trials was consistently significantly above chance 
values (t test; p < 0.001) in all task types. These trials were ex- 
cluded from the analyses described below. 


Basic task 

In this task, the monkeys discriminated the presence of corti- 
cal microstimulation (Fig. 2B). Although both monkeys were 
overtrained in the reaching task cued by arm vibration, their 
initial performance in the basic task cued by similarly pat- 
terned microstimulation was rather poor. This was despite the fact 
that the week before the start of the microstimulation cueing sessions 
they performed the task in which vibration was used as a cue with 
a 0.84 average performance on random trials for both monkeys 1 
and 2. 

In the initial sessions of the basic cortical microstimulation 
task, the behavioral performance was close to chance level. How- 
ever, both monkeys exhibited steady improvement in perfor- 
mance over the course of 40 sessions, eventually reaching levels 
that were significantly above chance (Fig. 4) (t test; p < 0.001). 
monkey 1’s learning curve was well fit with a linear regression 
model (Eq. 1) with an initial performance (intercept) of 0.58 and 
a learning rate (slope) of 0.009 increase in performance per ses- 
sion (r* = 0.75; p < 0.001). monkey 2’s learning curve was also 
well fit with a linear regression model (intercept of 0.46; learning 
rate of 0.010; r?7 = 0.77; p < 0.001). Note that monkey 1’s perfor- 
mance surpassed the chance level during the very first session, 
whereas monkey 2 remained at approximately chance level for 
the first 18 experimental days. This initial phase with little per- 
ceptible learning was followed by a large, single jump of 0.15- 
0.20 in performance. Significantly, once monkey 2 learned to use 
microstimulation cues in the basic task, it was able to rapidly 
learn more challenging discrimination tasks in the experiments 
that followed. 


Psychometric curve 

To study the range of microstimulation amplitudes that could be 
used to deliver reliable sensory input to the brain, we examined 
the performance of these monkeys on the well-learned basic task 
as we altered the amplitude of intracortical microstimulation. 
The relationship between current pulse amplitude and perfor- 
mance was well described by a sigmoidal psychometric curve (Eq. 
2) for each monkey with goodness of fit (r*) of 0.93 and 0.76 for 
monkeys 1 and 2, respectively (Fig. 5). The respective sigmoidal 
fits were 


0.49 
= 1 4 7 0:05(*=72) +0.48 (3) 
and 
0.34 
ya 1 + e004 x60) + 0.44. (4) 
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Figure 4. Improvement in discrimination performance (green) in the basic task in which 
monkey 1 (A) and monkey 2 (B) detected the presence of microstimulation and reached toward 
the target contralateral to the stimulated left hemisphere. Linear interpolation lines and their 
equations (red) are shown for monkeys 1 and 2, and for monkey 2, a sigmoidal interpolation line 
and its equation (blue) are shown. Chance performance level is 0.5 (brown). 


The thresholds, defined here as the sigmoidal inflection point, 
were 72 and 60 wA for monkeys | and 2, respectively, with 90% 
of maximum performance being achieved at 109 and 
117 pA (Fig. 5). 


Task reversal 


Ina subsequent set of sessions, we investigated whether the mon- 
keys could learn a new behavioral rule instructed by the same 
pattern of microstimulation (Fig. 2C). Although monkey 1 exhibited 
steady learning over the total course of the sessions, monkey 2’s 
learning was characterized by a rapid improvement, followed by 
slower gradual enhancement of performance. Monkey 1 (intercept, 
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tionship between behavioral performance (green) quality and microstimulation amplitude. The 
curves were constructed by interpolating behavioral data with sigmoidal curves (blue). Chance 
performance level is 0.5 (brown). 


0.29; slope, 0.064; r? = 0.97; p < 0.001) and monkey 2 (intercept, 
0.67; slope, 0.025; r7 = 0.50; p < 0.05) each exhibited linear learning 
curves (Fig. 6). The rapid jump in performance for monkey 2 is 
reflected in the higher intercept and lower slope. For both monkeys, 
high performance levels were reached over a much shorter time 
course compared with original learning of the basic task (0.064 and 
0.025 slopes for reversal learning vs 0.008 and 0.010 for the original 
task for monkeys 1 and 2, respectively). 


Temporal pattern discrimination 

In the next set of experimental sessions (Fig. 2 D), we investigated 
whether the same monkeys could discriminate the temporal 
characteristics of intracortical microstimulation, in addition to 
merely being able to detect the presence of stimulation. Learning 
for monkey 1 (intercept, 0.77; slope, 0.012; r= 0.66; p < 0.005) 
and monkey 2 (intercept, 0.70; slope, 0.020; p= 0.70; p < 0.001) 
proceeded steadily, with initial performance above chance but 
below final performance on the basic task (Fig. 7). Again, high 
performance levels were reached over a shorter time course than 


Fitzsimmons et al. e Reaching Cued by Microstimulation 
Rule Reversal Task 


A Monkey 1 


y = 0.064x + 0.29 
r? = 0.97 


Fraction Correct 


1 2 3 4 5 6 7 8 9 10 
Session 
Monkey 2 
Boos 
09 
0.85 


y = 0.025x + 0.67 
r? =0.50 


Fraction Correct 
°o 
~s 
uo 


Basic Task Learning Rate = 0.010 


0.5 
OS pm TST ee eee 
Chance Level 
0.45 
1 2 3 4 5 6 7 8 9 10 
Session 
Figure 6. —|mprovementin discrimination performance (green) of reversal task [i.e., the task 
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the basic task]. Linear interpolation curves and their equations (red) are shown. For comparison, 
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that of initial learning in the basic task (0.85 performance level 
reached on the seventh and eighth sessions for temporal pattern 
learning vs the 32nd and 39th sessions for the original task for 
monkeys 1 and 2, respectively). 


Spatiotemporal pattern discrimination 

In the final set of experimental sessions (Fig. 2 E), we investigated 
whether the owl monkeys could discriminate the spatiotemporal 
characteristics of intracortical microstimulation delivered 
through multiple implanted electrodes. Learning for monkey 1 
(intercept, 0.85; slope, 0.011; r? = 0.36; p < 0.001) and monkey 2 
(intercept, 0.77; slope, 0.020; r? = 0.65; p < 0.001) reached high 
values during the very first experimental session and proceeded 
steadily afterward. In this task, the learning reached a plateau of 
saturated performance after approximately the third or fourth 
sessions. Because of such quick saturation, a sigmoidal model- 
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monkey 1 (A) and monkey 2 (B). Linear interpolation curves and equations (red) are shown. 
Chance performance level is 0.5 (brown). 


proved to be a better fit in both cases (Fig. 8). The respective 
sigmoidal fits for monkeys 1 and 2 were 


0.16 
= L + e274 4-2.06) 


+ 0.78 (1? = 0.87) (5) 


and 


106.12 
aa Lt e444 13.25) 


105.18 (1? = 0.76). (6) 


High performance levels were reached over a shorter time course 
than that of initial learning in the previous tasks (0.85 perfor- 
mance level reached on the second and third sessions for spatio- 
temporal pattern learning vs the 32nd and 39th sessions for the 
original task and the seventh and eighth sessions for temporal 
pattern learning for monkeys 1 and 2, respectively). In monkey 1, 
there was significant intrasession learning during the first three 
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for monkey 1 (A) and monkey 2 (B). Linear (red) and sigmoidal (blue) interpolation curves and 
equations are shown. Chance performance level is 0.5 (brown). 


sessions, with performance during the second half of each session 
significantly above the performance during the first half of the 
session ( p < 0.05). 

During several experimental sessions in which the monkeys 
discriminated spatiotemporal patterns of microstimulation, we 
also tested whether neuronal ensemble activity could be recorded 
while the microstimulation protocol was run (data not shown). 
The chronically implanted electrode arrays that were not being 
used for microstimulation were connected to a Plexon recording 
and spike discrimination system. Neuronal waveforms on non- 
stimulated arrays remained of the same quality as they were pre- 
viously, before a microstimulation device being connected. Each 
microstimulation pulse produced an artifact of a distinct shape, 
the amplitude of which was approximately threefold or more 
higher compared with neuronal spikes. However, the custom 
stimulation hardware that is used minimizes the duration of each 
stimulus artifact, ~1 ms. During that 1 ms, neuronal data cannot 
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be recorded with our current technology, but when stimulating at 
100 Hz, the frequency used in these studies, only 10% of the 
neuronal data is lost to the stimulation artifact. The remaining 
90% can be recorded normally and used to predict behavioral 
parameters. This technological achievement is important for fur- 
ther development of brain—machine interfaces with microstimu- 
lation feedback. 


Discussion 

In this study, we demonstrated that spatiotemporal patterns of 
microstimulation delivered to $1 through chronically implanted 
microelectrode arrays can convey useful information to the 
brain. Our experiments spanned several months. The monkeys 
progressed from learning to merely detect the presence of micro- 
stimulation to discriminating spatiotemporal patterns. The more 
they practiced, the easier it became for them to interpret novel 
microstimulation patterns. These results suggest that multichan- 
nel microstimulation can benefit neuroprosthetic applications 
that require sensory feedback. 


Methodological issues 

This study, along with previous ones from our laboratory (Wess- 
berg et al., 2000; Sandler, 2005), showed certain benefits of using 
owl monkeys in the experiments that require long-term testing 
using chronically implanted microelectrodes. The first is the sta- 
bility of the cortical implants. monkey 1 was implanted in 2003, 
and monkey 2 was implanted in 2001. Both monkeys retained 
good quality of recordings from their electrodes (Sandler, 2005), 
and their S1 electrodes remained effective for microstimulation. 
Long-term viability of neuronal recordings combined with the 
long-term efficiency of microstimulation makes owl monkeys a 
very useful species for developing neuroprosthetics for both ex- 
tracting information from the brain and delivering information 
to the brain (Lebedev and Nicolelis, 2006). 

Although our owl monkeys started learning the tasks cued by 
microstimulation relatively slowly, this characteristic may be ac- 
tually used to the advantage of learning experiments: large 
amounts of neural and behavioral data can be collected over sev- 
eral recording sessions during which monkeys learn (Sandler, 
2005). The stability of implanted electrodes assures the consis- 
tency of both neuronal recordings and microstimulation. 


Does microstimulation evoke perception? 
Numerous publications have raised the question of whether elec- 
trical microstimulation of sensory areas evokes perceptions 
(Romo et al., 1998, 2000, 2002; Liu and Newsome, 2000; Ohara et 
al., 2004; Otto et al., 2005; Tehovnik and Slocum, 2005). Romo et 
al. (1998), who studied single-channel acute microstimulation of 
S1 in rhesus monkeys, made a strong argument in favor of per- 
ceptions evoked by microstimulation being similar to those 
evoked by skin vibrotactile stimulation. Their rhesus monkeys 
had no difficulties comparing sensations evoked by vibrotactile 
stimulation to those evoked by $1 microstimulation and detect- 
ing the instances in which the frequency of microstimulation 
pulses matched the frequency of vibration. 

The owl monkeys in our experiments initially did not perform 
as well as the rhesus monkeys in the study of Romo et al. (1998) 
after vibrotactile stimuli were replaced by microstimulation. This 
occurred despite the fact that both monkeys were previously 
overtrained in the reaching task cued by arm vibration. One pos- 
sible explanation for the inability of owl monkeys to generalize 
the previously learned task to the microstimulation cue is that the 
owl monkeys may have experienced vibration-like sensations on 
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their hands instead of their arms, where the mechanical vibrators 
had been placed previously. Another possibility is that micro- 
stimulation did not feel like vibration. In either case, the initially 
poor performance was probably related to the inability of owl 
monkeys to generalize the previously learned task to a somewhat 
different somatosensory cue. 

The difference between the previously used skin vibration and 
the microstimulation cue, however, did not prevent the monkeys 
from learning the task under the novel conditions. In addition to 
learning the basic task, they learned the reversal, temporal, and 
spatiotemporal discrimination tasks. Significantly, learning in 
each new task occurred much more rapidly compared with the 
previous one. Perhaps with long-term use of microstimulation, 
the sensation that it evoked became more vivid. In support of this 
suggestion, use of a tactile stimulation as a channel for artificial 
vision was reported to allow subjects to develop qualitatively new 
perceptions (Bach-y-Rita, 1983; Segond et al., 2005). 


Does microstimulation disrupt neural processing? 

Although the monkeys in our study learned to discriminate the 
patterns of multichannel microstimulation, electrical stimula- 
tion can be disruptive for neural processing, preventing, or bias- 
ing normal behavior. Cortical functioning can be disrupted by 
both transcranial magnetic stimulation (Valero-Cabre et al., 
2006) and microstimulation (Histed and Miller, 2006). We did 
not observe any deficits in motor performance or any bias to 
reach to a specific target after stimulation. We therefore conclude 
that disruption of neural processing, even if it affected the sensa- 
tion from the monkeys’ hands, did not have a role in the present 
task. Interference between continuous microstimulation (e.g., 
mimicking proprioception) and movements will have to be ad- 
dressed by additional studies. 


Stimulation parameters 

Our psychometric measurements demonstrated that owl mon- 
keys’ ability to discriminate microstimulation patterns is similar 
to that previously reported in the studies of discrimination ability 
of rhesus monkeys using peripheral stimuli (LaMotte and 
Mountcastle, 1975) and cortical microstimulation (Murasugi et 
al., 1993; de Lafuente and Romo, 2005). As in previous studies, 
owl monkey performance for different stimulus amplitudes was 
well described by a sigmoidal curve. The psychometric curves 
that we and others obtained can be recommended generally for 
choosing microstimulation parameters in neuroprosthetics that 
attempt to create an artificial somatosensory channel. 


Long-term effects 

Running the microstimulation sessions for many days did not 
result in any neurological problems in the monkeys or deteriora- 
tion in performance, which could be expected if microstimula- 
tion caused significant electrolytic damage to cortical tissue (Liu 
et al., 1999). On the contrary, with prolonged usage of micro- 
stimulation, performance steadily improved and eventually sur- 
passed the levels previously observed for vibrotactile cues. The 
arrays in S1 used for stimulation were able to record multiunit 
activity throughout the study. The recording quality at stimula- 
tion sites remained at the initial level. Electrode arrays not used 
for stimulation remained viable for higher quality recordings and 
showed no observable changes after long-term stimulation on the 
implanted S1 arrays. Such long-term stability makes $1 micro- 
stimulation a sound candidate for creating an artificial somato- 
sensory channel in neuroprosthetic applications. 


313 


Fitzsimmons et al. e Reaching Cued by Microstimulation 


Task reversal 

The ability of owl monkeys to reverse reach direction in the task 
cued by vibratory stimulation was demonstrated previously 
(Sandler, 2005). In this study, we showed that owl monkeys can 
do the same reversal when arm reach direction is cued by micro- 
stimulation. Such flexibility of behavioral response makes micro- 
stimulation a promising candidate for neurophysiological studies 
of rule representation and decision-making (Wise and Murray, 
2000; Muhammad et al., 2006). Indeed, the delivery of micro- 
stimulation pulses can be controlled precisely and the subsequent 
neural responses measured in different brain areas. In combina- 
tion with manipulations of behavioral rules imposed by the task, 
such an approach could be very productive for elucidating the 
neuronal computations between the stimulus arrival and the be- 
havioral response. Furthermore, by using microstimulation at 
different neural processing stages, more can be learned about the 
specific layout of cortical circuitry and processing. 


Discrimination of temporal patterns 

Our demonstration that owl monkeys can learn to discriminate 
temporal characteristics of microstimulation is in agreement 
with the ability of rhesus monkeys to discriminate the frequency 
of S1 microstimulation (Romo et al., 2002). However, it is note- 
worthy that the stimuli that Romo et al. (2002) used contained 
different numbers of microstimulation pulses (ie., different 
charges injected in the cortex), whereas in our experiments, the 
number of pulses was kept constant, which ensured that the mon- 
keys based their discrimination on the temporal pattern of the 
stimulus rather than on its intensity. 


Discrimination of spatiotemporal patterns 

In this study, we demonstrated for the first time that owl monkeys 
can learn to discriminate spatiotemporal characteristics of corti- 
cal microstimulation created using multiple implanted elec- 
trodes. This suggests that there is depth and complexity to the 
animals’ ability to “read out” microstimulation patterns. Adding 
the spatial dimension to the viable cortical microstimulation dis- 
crimination space significantly increases the potential through- 
put of information delivery to the brain. Indeed, by varying both 
temporal patterns of microstimulation and cortical sites to which 
it is delivered, as well as by taking advantage of stimulating se- 
lected cortical sites simultaneously, one can hope to significantly 
increase the amount of information conveyed to the brain in real 
time. Therefore, we suggest that multichannel microstimulation 
can be used in neuroprosthetic devices for delivery of complex, 
spatially, and temporally modulated signals to the brain. Infor- 
mation transfer rates that can be attained using microstimulation 
are of critical importance for neuroprosthetic application. Al- 
though our present estimation of the information capacity of 
multichannel microstimulation was limited by the modest learn- 
ing ability of owl monkeys, our results indicate that the micro- 
stimulation “vocabulary” is extendable. Indeed, the monkeys 
learned new microstimulation patterns with progressively im- 
proving ease. Therefore, we expect that these monkeys would be 
able to learn a vocabulary of at least 10 different spatiotemporal 
stimuli, a number that almost certainly can be, by far, exceeded by 
human users of sensorized neural prostheses. The bandwidth of 
spatiotemporal patterns suitable for information delivery to the 
brain will have to be explored in future studies, both in temporal 
and spatial dimensions. In this study, we used relatively long, 
stereotypic, and repetitive microstimulation trains. It is possible 
that the very initial portions of these trains were sufficient for 
successful discrimination. In practical applications, information 
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transfer rates could be increased by using much shorter micro- 
stimulation patterns with larger variability in their spatiotempo- 
ral structure. 


Encoding principles to be implemented 

To advance in the challenging tasks of replacing or providing new 
somatosensory sensations, we need a basic understanding of the 
responses that temporally patterned microstimulation produces 
in somatosensory cortical areas and the mechanisms by which the 
brain can use these responses to guide behavior. It would be naive 
to expect that neuronal activity patterns evoked by microstimu- 
lation would match normal somatosensory patterns. For in- 
stance, given the very complex excitatory and inhibitory effects 
produced by microstimulation (Butovas and Schwarz, 2003), it 
would be hard to reproduce rate-encoding features in $1 (Romo 
et al., 2002) using microstimulation. Increases in the frequency of 
microstimulation may not affect the total number of spikes firing 
in a predictable way, as excitatory periods begin to interfere with 
inhibitory periods. However, certain temporal patterns (Mount- 
castle et al., 1969, 1990; Lebedev et al., 1994) can be faithfully 
reproduced by cortical neurons. Indeed, during microstimula- 
tion, neuronal responses of cortical cells are clearly entrained to 
microstimulation pulses (Butovas and Schwarz, 2003), which 
provides a tool to deliver to the brain a variety of temporally 
patterned signals. Although the effect of microstimulation on 
average neuronal rates is somewhat unpredictable because of the 
aforementioned excitatory—inhibitory interactions (Butovas and 
Schwarz, 2003), it is more certain that groups of neurons in the 
stimulated area would entrain their activity to microstimulation 
patterns. Such synchronous, temporally patterned activity can 
then propagate to subsequent processing stages in the brain 
(Tehovnik et al., 2006). Although it would probably be impossi- 
ble to replicate naturally occurring neural activity with current 
pulses emanating from metal electrodes inserted into the brain, 
the fidelity of transmission from electronic transducers could be 
sufficient for a variety of neuroprosthetic applications and thus 
enhance the quality of life for users of those devices in ways both 
measurable and immeasurable. 
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Background. During planning and execution of reaching movements, the activity of cortical motor neurons is modulated by 
a diversity of motor, sensory, and cognitive signals. Brain-machine interfaces (BMls) extract part of these modulations to 
directly control artificial actuators. However, cortical modulations that emerge in the novel context of operating the BMI are 
poorly understood. Methodology /Principal Findings. Here we analyzed the changes in neuronal modulations that occurred 
in different cortical motor areas as monkeys learned to use a BMI to control reaching movements. Using spike-train analysis 
methods we demonstrate that the modulations of the firing-rates of cortical neurons increased abruptly after the monkeys 
started operating the BMI. Regression analysis revealed that these enhanced modulations were not correlated with the 
kinematics of the movement. The initial enhancement in firing rate modulations declined gradually with subsequent training 
in parallel with the improvement in behavioral performance. Conclusions /Significance. We conclude that the enhanced 
modulations are related to computational tasks that are significant especially in novel motor contexts. Although the function 
and neuronal mechanism of the enhanced cortical modulations are open for further inquiries, we discuss their potential role in 
processing execution errors and representing corrective or explorative activity. These representations are expected to 
contribute to the formation of internal models of the external actuator and their decoding may facilitate BMI improvement. 
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INTRODUCTION 


Brain Machine Interfaces (BMIs) hold promise for restoring motor 
functions in severely paralyzed patients [1-7]. State of the art 
BMIs take advantage of recent advances in electrophysiological 
techniques and neural decoding algorithms. Multi-electrode arrays 
facilitate simultaneous recordings from hundreds of neurons in 
multiple cortical areas. Movement related signals that modulate 
the activity of these neurons are extracted using neural decoding 
techniques and employed to control an external actuator. The 
BMI neural decoder includes free parameters that in typical BMI 
experiments with monkeys are determined from neural recordings 
during a training session with reaching movements. However, the 
requirement to produce similar movements through the BMI 
introduces a novel motor context, which may in turn affect the 
cortical modulations that drive the BMI. 

During planning and execution of reaching movements, the 
modulations in the firing rate of cortical motor neurons reflect 
multiple motor, sensory, and cognitive variables [8-12]. Neural 
modulations that represent the direction and speed of the 
movement have been extensively studied during stereotypical 
reaching movements, and described computationally using tuning 
curves [13-16]. Recent BMI experiments indicate that neural 
tuning to movement direction [6] or velocity [7,17] may change 
following BMI operation. However, changes in neuronal modula- 
tions beyond those related to movement kinematics have not been 
investigated. 

The operation of a BMI presents a novel motor context in 
which the external actuator is controlled based on the predictions 
generated from a limited subset of neurons. During initial BMI 
operation, the movement of the actuator may deviate from the 
intended movement, and result in errors, as evident from the 
degradation in behavioral performance [7]. Here we addressed the 
effect of operating in this novel motor context on the nature of 
neuronal modulations in the motor cortex. Our  spike-train 
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analyses show that initial BMI operation was associated with 
increasing neuronal modulations, which were not merely associ- 
ated with movement kinematics. 

Firing-rate modulations are masked by neural noise, which 
hampers their unambiguous estimation form recorded spike-trains 
[18]. Averaging techniques that are applied to reduce the neural 
noise also diminish the effect of rate-modulating signals. Current 
rate-estimation methods focus on firing-rate modulations that are 
correlated with specific modulating signals, like the direction and 
speed of the movement. Such methods rely on identifying the 
relevant modulating signals and ignore potential contributions by 
other signals [19]. Instead of estimating the firing-rate, we focus in 
this paper on quantifying its variance, i.e., the variance of the 
neural activity that is associated with overall rate-modulations. 
This provided a scale for assessing the variance associated with 
specific modulating signals, which were estimated using linear 
regression. Using these tools we demonstrate that the variance 
associated with neuronal rate-modulations increased during initial 
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BMI operation without a matching increase in the variance 
explained by the movement kinematics. Furthermore, the variance 
associated with neuronal modulations decreased with subsequent 
BMI training sessions. Possible hypothesis regarding the nature of 
these enhanced modulations are discussed to motivate further 
research. 


METHODS 
Behavioral task and brain-machine interface 


operations 

The BMI experiments were performed in two adult female 
monkeys (Macaca mulatta) and consisted of 10 BMI sessions for 
the first monkey and 23 for the second. The experiments are 
described in detail in [7] and briefly described here. Neural 
activity was recorded from N,= 100-300 neurons in multiple 
cortical areas including the primary motor cortex (M1), dorsal 
premotor cortex (PMd), supplementary motor area (SMA), and 
primary somatorsensory cortex (Sl) in one monkey and medial 
intraparietal (MIP) of the posterior parietal cortex (PP) in the 
second monkey. 

Each experimental session started with a training session in 
which the monkey controlled the position of a cursor on 
a computer screen by moving a hand-held pole (pole control), with 
the task of acquiring a randomly placed visual target within 5 sec 
to obtain a juice reward. A linear filter was trained to predict the 
velocity of the movement from the binned spike-counts of the 
recorded neurons. After training, the filter generated real-time 
predictions of the velocity, which were reproduced by the cursor 
and/or a robotic arm (brain control). The monkeys continued to 
move the pole after brain control started (brain control with hand 
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movements, BCWH), but later assumed a stationary arm posture 
after the pole was taken away (brain control without hand movements, 
BCWORH). Lack of muscle activity during BCWOH was demon- 
strated by EMG measurement from wrist flexors, extensors and 
biceps. Performance accuracy diminished after the transition from 
pole- to brain-control, and after the monkeys stopped moving their 
arms. However, the BMI task performance improved with further 
training in all the control modes, clearly indicating that motor 
learning was involved. 


Percent Overall Modulation (POM) 

Spike-trains can be considered as realizations of point processes 
20,21]. he number of spikes recorded in a bin of size 6, Nj, 
depends on the average bin-rate in that bin A,, which is 
modulated by the encoded signals, as depicted in Figure | (upper 
diagram). However this dependence is stochastic and the variance 
of the spike-count Var[.N,] may differ from the variance of the bin- 
rate Var[A,]. While the variance of the spike-count can be 
measured directly, it is the variance of the bin-rate that is of 
interest here because it reflects signal modulations. In order to 
investigate these modulations, we defined the percent overall 
modulation (POM) as: 


_ Var[ Aj) 


Since the variance of the bin-rate cannot be measured directly, the 
POM cannot be estimated without further assumptions. However, 
instead of restricting the analysis to firing-rate modulations that 
involve specific modulating signals, we made only basic assump- 
tions about the nature of the spikes trains. In particular, we 
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Figure 1. Inhomogeneous Poisson process and implied variance distribution. 


doi:10.1371/journal.pone.0000619.g001 
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assumed that the spike trains are realizations of inhomogeneous 
Poisson processes, which are the simplest point processes that can 
describe rate modulations. We further generalized the analysis to 
additive noise models whose variance is proportional to the mean. 

The simplest point process, the homogenous Poisson process, is 
characterized by a constant rate and thus is inadequate for 
describing task-related firmg rate modulations. Despite their 
constant rates, homogeneous Poisson processes generate spike 
trains with highly variable spike-counts, whose variance equals the 
mean, Le., Var[N;] = E[N;]. Taking the homogeneous Poisson 
process as a model of neural activity with no modulations, we 
conclude that the resulting variance, which equals the mean rate, 
is attributed to the neural noise. 

The mhomogencous Poisson process, which is characterized by 
time-varying rate that is independent of the history of the spike 
train, is the simplest point process that can describe firing rate 
modulations [20,21,22]. For inhomogeneous Poisson processes, 
the variance of the spike-count Var[.Nj] is related to the variance of 
the bin-rate Var[A;] according to Var[.N,] = Va[A,|+E[N] [20]. 
This relationship can be interpreted as a decomposition of the total 
variance in the binned spike-count into the variance of the 
underlying information bearing parameter, or bin-rate Var[A,], 
and the variance that would have occurred if Vy, was generated by 
a homogenous Poisson process, E[.N;], 1e., the neural noise 
(Figure 1). Thus, the variance of the modulated bin-rate is the 
excess variance of the binned spike-count beyond the level 
expected from a homogeneous Poisson process. The resulting 
POM can be evaluated from the statistics of the binned spike-count 
as (Figure 1): 


Pom = Vall ogy, = VarlNel— FINA ogy, = 
Var|No| Var|No| 2) 
1 
(1 — z) 100% 
Var|No| . : : ; 
where F = as | is the Fano factor [23,22]. Given spike-trains 
E|N5] 


of finite duration, we estimated the POM using the sample-mean 
and sample-variance of the binned spike-counts, instead of the 
mean and variance, respectively. 

The POM analysis can be extended to cases where the binned 
spike-count include  signal-dependent zero-mean noise, Le., 
N, = Ajte [24]. The probability distribution of the noise ¢ is assumed 
to be conditionally normal with signal dependent variance: 
fe\| A)~M0,y7A,). Note that this model converges to the in- 
homogeneous Poisson process for large rates and y= 1. Since for 
any signal level the error is zero mean, the variance of the spike 
counts is given by: Var[.Nj] = Var[A,]+y°E[Nj], as shown in the 
Supplemental text (Text $1). The resulting POM can be evaluated as: 


Var| Ap] 
Var| No} 


=(1—y’):100%+)°POM(1) 


Var|N] —y° E[No] 
Var| No] 


POM(y)= 100% = 100% 


where POM(1) is the POM of an inhomogeneous Poisson process 
(y = 1) as defined in Equation (2). Thus, the results and conclusions 
based on the POM defined in Equation (2) can be easily extended for 
the general case of additive signal-dependent noise. In particular: 
dPOM(y) 
dPOM(1) 
based on Equation (2) reflect imcreasing/decreasing trends in 
POM(y). 


=", so increasing/ decreasing trends in estimated POM 
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Significantly Modulated Neurons 

The POM of spike-trains that are evoked by homogeneous Poisson 
processes is zero. However, when estimated from finite spike- 
trains, the sample-POM is randomly distributed and may take non- 
zero, either positive or negative, values. Under the homogeneous 
Poisson process assumption, the distribution of the sample-POM, 
estimated from a finite sample of n bins, is asymptotically normal 
with zero mean and variance of 2/n. We defined neurons as 
significantly modulated if their estimated POM indicated that the 
homogeneous Poisson process hypothesis could be rejected at 95% 
confidence level for at least one control mode (pole, BCWH, or 
BCWORH). For a typical sample of 12000 bins (20 minutes of an 
experiment with 100 msec bins), the standard deviation of the 
sample- POM is 1.3%. Note that neurons that are significantly 
modulated in only one or two control modes may exhibit zero of 
even negative sample-POM in the other mode(s) due to the limited 
length of the analyzed spike-train. 


Percent Velocity Modulation (PVM) 

Velocity tuning is often determined in center-out reaching 
experiments, where the direction of movement is approximately 
constant [13,16]. For general arm movements, this method was 
generalized to characterize the tuning of the neural activity to the 
velocity at a specific time lag [17]: 


Nk)=a.(DV AKAD +a,DV (K+) +aDS(k +) +a(D + (4) 
(k,l) , 


where fis the index of the current time-bin, (A) is the spike counts 
(in bins of 100msec), V, and V, are the x- and y-components of the 


velocity, S=/ V+ V; is the speed, / is the relative lag (positive/ 


negative / corresponds to rate-modulations preceding/ succeeding 
the velocity measurement, respectively), a,(/), a,(d) and a,(/) are the 
tuning parameters, a,(/) is a bias parameter, and e(f, 2) is the 
residual error. The coefficient of determination of the single-lag 
regression R*(l) quantifies the fraction of the variance in the neural 
spike-count that is attributed to, or explained by, the velocity at lag 
1. However, since the velocity values at different lags are 
correlated, the lag-by-lag analysis cannot be used to determine 
the fraction of the variance that is attributed to the spatio-temporal 
velocity profile. Hence, we further generalized the analysis to 
account for the tuning of the neural activity to the spatio-temporal 
velocity profile in the surrounding window according to: 


Ly Ly 
N}= S> aVk+D+ S> a(DVi(k+)+ 
i=-L i=-L 
Z L L (5) 
S> as) Sk +) +4e+ek) 
/=—-L 


where &(f) is the residual error, and L; and Ly are the number of 
preceding and succeeding lags in the velocity profile, respectively. 
We used L; = Ly = L=9 to include a 1900-ms window around the 
current bin. The regression in Equation (5) was evaluated using 
truncated Singular Value Decomposition to stabilize the solution 
despite the large condition number of the spatio-temporal velocity 
matrix (around 10°) [25,26]. Truncation was performed at the 
singular value that preserved 95% of the variance in the velocity 
measurements. 

The coefficient of determination of the spatio-temporal re- 
gression of Equation (5), R’, describes the fraction of the variance 
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in the binned spike-count that is linearly related to variations in the 
spatio-temporal velocity profile in the surrounding window. 
Expressed as a percentage, R° is referred to as the percent 
velocity-related modulation, or PVM. 


Percent Kinematics Modulation (PKM) 

The above analysis was further extended to include other 
kinematics variables that may modulate the neural activity, 
including the position and magnitude of acceleration: 


Ly I, 
NKH= S> bOPA+D+ So byOPK+D+ 
l=—-L, l=-L, 
Ly : Ly : ) 
S> bMSK+D+ So daDAK +1 +b. + ek) 
l=-L, l=-L 


Where P, and P, are the x- and y-components of the position, and 


A=,/ A+ A} is the magnitude of the acceleration vector whose 


x-component is given by 4,()) = V.)—V(i-1)= P()-2PU- 1) 
P(/—2) and y-component given by the corresponding expression. 
Note that the velocity and acceleration vectors are implicitly 
included in Equation (6) as the first and second order differences 
between the positions in consecutive lags. 

The coefficient of determination of the spatio-temporal re- 
gression in Equation (6) was expressed as a percentage and 
referred to as the percent kinematics modulation, or PAM. 


Variations in percent modulations 

In order to evaluate the variations in the percent modulations (Le., 
POM, PVM or PKM) within and across different control modes and 
training sessions, we estimated their values for each neuron in 2- 
minute non-overlapping intervals. Within each control mode, the 
POM, PVM and PKM of individual neurons were determined by 
averaging across all the relevant 2-minute intervals. ‘The ensemble- 
POM, ensemble-PVM and ensemble-PAM in a specific 2-minute 
interval were determined by averaging across all the significantly 
modulated neurons. Comparisons between control modes and 
training sessions were based on the mean-POM, mean-PVM and 
mean-PAM, which were computed by averaging the correspond- 
ing ensemble values across all the intervals in the same control 
mode. 


Principal Component Analysis 

Principal component analysis (PCA) is a standard technique for 
discovering the dimensionality of a data set and decomposing it 
into uncorrelated components [27,28]. The sequences of spike- 
counts from WV, neurons is transformed linearly (with unit norm 
weight vectors) into WV, principal components, which are un- 
correlated with each other and have extreme variance values. 
Since the principal components in this application are weighted 
sums of the recorded spike-counts they may be referred to as 
“principal-neurons’’. 

Specifically, we computed the N,N, sample-correlation matrix 
of the normalized spike-counts of the WV, neurons, whose 77 
element is the sample-correlation between the normalized spike- 
counts N; and N; of the 7-th and j-th neurons, recorded during the 
relevant part of the experiment. The variance of the principal- 
neurons was determined by the eigen-values 21 >/2> --- > An, of 
the sample-correlation matrix. In particular, 2, is the maximum 
variance of any weighted-combination of the recorded spike- 
counts with a unit norm weight vector. The normalized eigen- 
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A hi : ; : : 
value A4;= 4 — ‘— describes the fraction of the variance in the 
yj 
j=l 
original data that is captured by the 7¢h principal-neuron. 
Expressed as a percentage, it defines the percent of the total 


variance in the neural ensemble carried by the 7-th principal- 
neuron. 


RESULTS 


The neural activity of most of the cortical neurons was more 
variable during brain control than during pole control. Generally, 
rate variability was higher than would be predicted by a homoge- 
neous Poisson process, as indicated by variance that exceeded the 
mean (Figure 2, top panels). The variance of the binned spike- 
count (in 100-ms bins) exceeded the mean spike-count for most of 
the N,, = 183 neurons recorded in this session, (83%, 88% and 92% 
during pole control, BCWH and BCWOKH, respectively). ‘This 
excess variability was also evident in the ratio of the variance to the 
mean (the Fano factor), which was found to be mostly above 1.0 
(Figure 2, bottom panels). Most importantly, the transition to brain 
control resulted in even higher excess variability and larger Fano 
factor for most of the neurons (78% and 87% of the recorded 
cortical neurons during BGWH and BCWOKH, respectively) as 
evident from the scatter plots of Figure 2. 


Percent Overall Modulation 

The percent overall modulation (POM), defined in Equation (1), 
represents the percentage of the variance of the binned spike-count 
that is attributed to rate modulations (Figure 1, Methods). ‘The pie- 
plots in Figure 3 illustrate the distribution of the variance in 
the spike-counts recorded from two M1 neurons in different 
control modes, based on the POM computed using Equation (2) 
(Methods). During pole control (Figure 3, top pies), rate 
modulations accounted for only 7% for the typical neuron 
depicted on the right and 36% for a highly modulated neuron 
depicted on the left. The contribution of rate-modulations to the 
variance of the neural activity was more significant in brain control 
with and without hand movements, accounting, respectively, for 
13% and 34% of the variance for the typical neuron (Figure 3, 
right middle and bottom pies) and 37% and 58% for the highly 
modulated neuron (Figure 3, left middle and bottom pies). These 
examples indicate that POM was higher in brain control than in 
pole control. 

Figure 4 demonstrates the same trend for all the significantly 
modulated neurons. Insignificantly modulated neurons, Le., 
neurons with little or no rate modulation in all the control modes 
(Methods), were considered irrelevant for task performance, and 
were excluded from further analysis. During the session analyzed 
in Figure 4, 87% of the 183 recorded neurons exhibited significant 
modulations in at least one control mode. Most of the significantly 
modulated neurons exhibited higher POM during brain control 
than during pole control, as evident in the top panels of Figure 4 
(78% and 91% of the significantly modulated neurons in BOWH 
and BCWOH, respectively, Table 1). For few neurons the 
estimated POM during pole control was even negative (possibly 
due to the finite length of the spike-train, see Methods) and 
became positive only during BOWH or BCWOH. Thus, the top 
panels in Figure 4 depict the changes in the firing rate statistics 
expressed in Figure 2 in terms of the POM. 

The distributions of the POM of the significantly modulated 
neurons during brain control (BCWH and BCWOH) differed 
significantly from the distribution of the POM during pole control 
(Wilcoxon rank sum test, p= 1.2% and p=107!°% for BCWH 
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Figure 2. Firing rate statistics for N,,= 183 units recorded in one session. Mean-variance relationship computed based on 100-ms bins (top panels); 
and scatter plots of Fano factor in brain versus pole control (bottom panels). BCWH-Brain control with hand movements, BCWOH-Brain control 


without hand movements. 
doi:10.1371/journal.pone.0000619.g002 


and BGWOH, respectively) as shown in the bottom left panel of 
Figure 4. The mean POM (+std) in the different modes were 
8.8%+10.7%, 12.2% +11.7%, and 18.8%+14.4% for pole 
control, BOWH, and BCWOH, respectively. The corresponding 
distributions for M1 and PMd neurons are shown in the right 
bottom panels of Figure 4. 

Variations in POM during that session are shown in Figure 5 for 
the ensemble-mean (top left) and for a typical PMd neuron (top 
right). The ensemble-POM is the average POM over all the 
significantly modulated neurons (Methods). The figure indicates 
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Figure 3. Percent Overall Modulation (POM) in pole control, BCWH 
and BCWOH for two representative M1 units. 
doi:10.1371/journal.pone.0000619.g003 
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that the ensemble-POM remained relatively stable in each mode of 
operation, but changed abruptly after the transition to brain 
control and especially after the transition to brain control without 
hand movements. 


POM and firing rate statistics 

The underlying changes in the spike-count statistics (bottom panels 
of Figure 5) indicate that the ensemble-POM increased mainly due 
to an increase in the variance of the spike-count, which was not 
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Figure 4. Percent Overall Modulation (POM) for N.j,=160 signifi- 
cantly modulated units recorded in the same session analyzed in 
Figure 2. Top: Scatter plot of POM in brain versus pole control. Bottom: 
distribution of POM of all significantly modulated units (left) and those 
in M1 (right upper, N,jg =56) and PMd (right lower, Nig = 55). 
doi:10.1371/journal.pone.0000619.g004 
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Table 1. Number of neurons with increasing POM (POM,,,,) 

: and increasing variance (Var;,.), during the sessions depicted 
: in Figure 4, and 5, out of the N,j, significantly modulate 

> neurons. 


: Transition All PMd M1 $1 SMA 

: Figure 4, 5 

2 Ny =183 Nig 160(*) 55 56 33 13 

: Pole  BCWH POMine 125 48 35 28 12 

: % of Nsig 78% 87% 62% 85% 92% 
Varine 97 44 33 «16 «4 

: % of POMinc 78% 92% 94% 57% 33% 

: BCWH > BCWOH POMin. 139 46 51 31 8 

: % of Neig 87% 84% 91% 94% 62% 
Varine 72 39 233 7 

: % of POMinc 51% 85% 45% 10% 87% 

: Pole > BCWOH — POMinc 145 51 50 32 #10 

: % of Neig 91% 93% 89% 97% 77% 
Varine 104 50 37. «12”—~C*~“‘ 

% of POMinc 72% 98% 74% 37% 50% 

: Figure 6 

> N,=185 Nig 169(*) 59 56 36 15 

: Pole—> BCWOH = POMine 155 56 50 35 11 

: % of Neig 92% 95% 89% 97% 73% 
Varine 116 52 36 #23 «(4 
% of POMinc 75% 93% 72% 66% 36% 


: (*) Including neurons in ipsilateral M1, which are too few to analyze as a group. 
: doi:10.1371/journal.pone.0000619.t001 


matched by the change in the mean spike-count. Indeed, most of 
the neurons (78%) whose POM was higher in BOWH than in pole 
control, exhibited also a larger variance, especially neurons in 
PMd (92%) and M1 (94%), as summarized in Table 1. 
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Figure 5. Variations in POM (top) and spike-count statistics (bottom) 
during the same experimental session analyzed in Figure 4 
(ensemble-average-left panels; single PMd unit-right panels). The 
POM and spike-count statistics were computed in 2-minute non- 
overlapping intervals and averaged across all the N.jg = 160 significantly 
modulated neurons to obtain the ensemble-average. 
doi:10.1371/journal.pone.0000619.g005 
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The transition from BCWH to BCWOH resulted in further 
increase in POM for most of the significantly modulated neurons 
(87%). This increase resulted from a combination of increasing 
variance and decreasing mean rate. Increasing POM from BCWH 
to BCWOH was usually associated with increasing variance, 
especially for PMd neurons (85%, Table 1), as shown for a typical 
PMd neuron in Figure 5 (bottom right). Furthermore, considering 
the overall transition from pole control to BCWOH, increasing 
POM was associated with increasing variance for most of the 
neurons (72%); especially among PMd (98%) and M1 (74%) 
neurons (Table 1). Thus, the overall increase in POM from pole 
control to BEWOH is attributed mainly to increasing variance. 

Direct transition from pole control to BOWOH was tested in 
later sessions in which the joystick was removed immediately after 
the pole control epoch. The changes in the POM and the firing 
rate statistics during such a session are depicted in Figure 6. The 
large increase in the POM after the transition to BOWOH (top 
panels, ensemble average—left, a typical PMd neuron-right) is 
related mainly to increasing variance (bottom panels, respectively). 
Indeed, most (75%) of the neurons whose POM was higher in 
BCWORH than in pole control, exhibited also a larger variance, 
especially neurons in PMd (93%), and M1 (72%) (Table 1). 


POM in different cortical areas 

The mean-POM, computed as the mean of the ensemble-POM 
across each control mode (Methods), was higher in brain control 
than in pole control for all cortical areas examined from both 
monkeys (Figure 7, Table 2). The standard deviation of the 
ensemble-POM during each control mode is marked by an error 
bar (reflecting variations in the ensemble-POM across time). ‘The 
change in the mean-POM from pole control to either BOWH or 
BCWOH was higher than the respective standard deviations in 
most of the recorded cortical areas (except for the limited group of 
SMA neurons). 


Variance distribution in principal neurons 
The POM analysis decomposed the variance of the spike-counts of 
individual neurons into the variance associated with rate 
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Figure 6. Variations in POM (top) and spike-count statistics (bottom) 
during an experimental session with direct transition form pole 
control to BCWOH (ensemble-average-left panels; single PMd unit- 
right panels). The POM and spike-count statistics were computed in 2- 
minute non-overlapping intervals and averaged across all the N,jg= 169 
significantly modulated neurons to obtain the ensemble-average. 
doi:10.1371/journal.pone.0000619.g006 
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Figure 7. Mean-POM of the N.jg significantly modulated units in 
different cortical regions during different control modes in a single 
session with each monkey. Full bars mark the mean values across all the 
2-minute intervals in the same control mode, and error bars represent the 
standard deviation. M1-primary motor cortex, PMd-dorsal premotor 
cortex, S1-primary somatosensory cortex, SMA-supplementary motor 
area, PP-Posterior Parietal Cortex (medial intraparietal). 
doi:10.1371/journal.pone.0000619.g007 


modulations and the variance associated with neural noise 
(Figure 3). At the ensemble level, principal component analysis 
(PCA, Methods) was used to decompose the total variance of the 
neural activity into uncorrelated principal components, or 
“principal neurons”. Figure 8 depicts the percent variance carried 
by the different principal neurons during the same session 
analyzed in Figures 4, 5 and 7. The percent variance dropped 
sharply for the first few principal neurons and remained relatively 
constant thereafter. In an ideal case, when the noise is generated 
from independent identically distributed random processes, the 
percent variance of the background noise is constant. Thus, the 
relatively constant variance level beyond the initial 5-15 principal 
neurons can be attributed mainly to the neural noise. In contrast, 
the excess variance of the initial principal neurons above the 
background level reflects correlated activity among the different 
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Figure 8. Percent variance carried by the different principal neurons 
during the same session analyzed in Figure 5. The analysis 
decomposed the variance carried by the ensemble of the significantly 
modulated neurons recorded from all brain regions. Inset: zoomed-in 
version of the same plot for the initial 20 principle neurons. 
doi:10.1371/journal.pone.0000619.g008 


neurons and can be attributed mainly to rate-modulations by 
common modulating signals. 

The decomposition of the variance of the ensemble activity into 
principal neurons extends the single-neuron based POM analysis, 
and suggests that the modulating signals responsible for the POM 
were highly correlated. Indeed, the accumulated excess variance in 
the initial principal neurons (Figure 8) was comparable to the 
average POM indicated in Figure 5, and small compared with the 
accumulated variance of the background noise level. Furthermore, 
as seen from Figure 8, the variance of the first principal neuron, 
i.e., the maximum variance of any weighted sum of the neural 
activity (with unit norm weight vector), was higher during brain 
control than during pole control. This indicates that the variance 
of the modulating signals, which contributed to correlated neural 
activity, was higher during brain control than during pole control. 


Percent velocity modulation 

The contribution of the velocity of movement (both the velocity- 
vector and the speed) to the variability in the neural activity, was 
evaluated using the regression between the spike counts in 100-ms 


Table 2. Mean-POM and Mean-PVM of neurons in different brain regions during each control mode. The standard deviations reflect 


: variations in the ensemble-statistics across time. 


> Mean+std PMd 


M1 S1 SMA PP 
Monkey #1 #1 #1 #1 #2 
Nig 55 56 33 13 62 
POM (%) Fig 7 Pole 2.9+0.9 17.5+0.7 5.7+1.0 6.6+2.3 21.5+1.0 
BCWH 7541.1 20.0+1.1 8.7+0.9 10.1+2.7 25.7+3.3 
BCWOH 12.0+0.7 30.1+1.6 15.5+1.5 10.7+2.9 27.6+2.9 
PVM (%) Fig 10 Pole 1.2+0.2 7.2+0.5 3.7+0.7 43+14 7.3+0.6 
: BCWH 2.3+0.3 6.4+0.5 3.3+0.5 4.2+0.9 9.8+2.6 
BCWOH 2.1+0.3 5.1411 2.7+1.0 2.4+0.4 6.9+3.6 
PKM (%) Pole 1,.3+0.2 8.0+0.7 4.2+0.8 4441.3 7.2+0.6 
; BCWH 2.6+0.4 7.140.5 3.9+0.5 4.3+0.9 9.2+2.6 
BCWOH 2.4+0.3 5.8+1.2 3.2411 2.6+0.5 7.2+3.8 


: doi:10.1371/journal.pone.0000619.t002 
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bins and the velocity profile during the surrounding 1900-ms 
window (Methods, Equation (5)). Spike trains evoked during pole 
control were related to the hand velocity, while spike trains evoked 
during BCWOH were related to the cursor velocity. The analysis 
for BGWH accounted for the correlation with both the hand and 
cursor velocities. The coefficient of determination, R®, of the 
regression quantifies the fraction of variance in binned spike- 
counts that is attributed to the velocity profile. Expressed as 
a percentage, the coefficient of determination 1s referred to as the 
percent velocity-related modulation (PVM, Methods). 

We also extended the analysis to include all the kinematics 
variables, i.e., the time course of the position (which implicitly 
includes the time course of the velocity and acceleration), speed 
and magnitude of the acceleration, in the surrounding 1900-ms 
window, as described by Equation (6) (Methods). The resulting 
coefficient of determination was expressed as a percentage and 
referred to as percent kinematics modulation (PAM, see Methods). 
Overall, the results for the PAM were similar to the PVM, and are 
stated only briefly for completeness. 

PVM was significantly correlated with POM (p<107'°), as 
shown in Figure 9, with coefficients of correlation of 0.72, 0.68 and 
0.79 for pole control, BOWH and BCWOKH, respectively (0.72, 
0.71 and 0.81, respectively for PAM versus POM). This strong 
correlation implies that the activity of cortical neurons, which 
exhibited larger rate modulations, was, in general, better 
correlated with the velocity profile, and can thus contribute more 
to its prediction. In contrast, there was no significant correlation 
between POM, or the increase in POM, and the magnitude of the 
decoding weights used in the BMI filter. 

While PVM was strongly correlated with POM, the slope of 
the lear relationship was only 0.31, 0.21 and 0.12, for pole control, 
BCWH and BCWOKH, respectively (0.35, 0.23 and 0.14, respec- 
tively for PAM versus POM). The smaller than unit slopes indicate 


POM (%) 


Figure 9. Correlation between PVM and POM for all the N.j,=160 
significantly modulated neurons recorded in one session. Coloured 
dashed lines mark the linear regression lines. Black dashed lines mark 
the diagonal unit-slope lines. 

doi:10.1371/journal.pone.0000619.g009 
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that only a small fraction of the modulations was correlated with the 
velocity profile. Most importantly, the slopes of the PVM-POM or 
PKM-POM relationships were smaller in brain control compared to 
pole control. This suggests that modulating signals, which were not 
correlated with the kmematics of the movement, had a larger effect 
on the POM during brain control than during pole control. 

The mean-PVM in most cortical regions did not increase 
significantly after switching to brain control, as indicated in 
Figure 10. Thus the mean-PVM did not follow the significant and 
large increase in the mean-POM shown in Figure 7. While the 
mean-POM of M1 neurons, for example, increased from 
17.5+0.7% in pole control to 20.041.1% im BCWH and 
30.1£1.6 in BCWOH, the mean-PVM decreased from 
7.2+0.5% to 6.4£0.5% and 5.1£1.1% (Table 2). The mean- 
PKM (Table 2) was at the most 0.8% higher than the mean-PVM, 
and decreased from 8.0+0.7% for M1 neurons during pole control 
to 7.14£0.5% and 5.841.2%, during BCWH and BCWOH, 
respectively. Since the increase in POM was not matched by 
increasing PVM or PKM, the higher neuronal rate modulations 
observed durmg brain control cannot be explained only by 
increased modulations due to the kinematics of the movement. 


Effect of training 

In all the experimental sessions, with both monkeys, the POM was 
always higher in brain control than im pole control, as 
demonstrated in Figure 11 (top panel). Furthermore, in all the 
control modes, the POM decreased gradually with training. These 
trends were statistically significant in pole control and BCWH 
(p<0.05). In contrast, the mean-PVM remained approximately the 
same, and even increased, with training (Figure 11, middle panel). 
The difference between the two, 1.e., POM minus PVM, 1s depicted 
in the bottom panel of Figure 11, and exhibited statistically 
significant decreasing trends in all the control modes (p<0.02). 
Similar results were obtained when considering the mean-PAM, 
which accounts for modulations by the kinematics of the 
movement. Thus, the changes in POM during BMI training seem 
to reflect mainly changes in untagged modulations not correlated 
with the kinematics of the movement. 
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Figure 10. Mean-PVM of the N.i, significantly modulated units in 
different cortical regions during different control modes in the same 
sessions as Figure 7. Full bars mark the mean values across all the 2- 
minute intervals in the same control mode, and error bars represent the 
standard deviation. M1-primary motor cortex, PMd-dorsal premotor 
cortex, $1-primary somatosensory cortex, SMA-supplementary motor 
area, PP-posterior parietal cortex (medial intraparietal). 
doi:10.1371/journal.pone.0000619.g010 
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Session 


Figure 11. Effect of training: percent overall modulation POM, 
percent velocity modulation PVM and percent untagged modulation 
PUM of the N,j, significantly modulated neurons during all the 10 
BMI sessions with monkey #1. N.ig= 156, 130, 142, 142, 150, 140, 160, 
166, 174 and 165. Initial sessions included BCWH while later sessions 
included only BCWOH. 

doi:10.1371/journal.pone.0000619.g011 


The observed trends in the mean-POM and in the percent 
untagged modulations mirrored the trend in task performance [7]. 
Behavioral performance, quantified by either the time to reach the 
target or the success-rate, deteriorated after switching to brain 
control, but improved gradually with training. Thus, in general, 
the POM and the percent untagged modulations increased as the 
behavioral performance degraded and decreased as the behavioral 
performance improved. 


DISCUSSION 


Our analyses indicate that cortical neurons that are used to control 
a BMI modulate their activity more intensely during brain control 
than during pole control. The enhanced modulations were evident 
in increased variability of the binned spike-counts beyond the level 
expected from homogeneous Poisson processes. We quantified this 
excess variability using the POM, which described the overall 
contribution of modulating signals to the variance of the binned 
spike-counts of individual neurons. The method for estimating the 
POM was based on the assumption that the spike trains were 
realizations of inhomogeneous Poisson processes. Nevertheless, we 
showed that the analysis is relevant for a broader class of spike- 
trains with signal dependent additive noise. Thus, the observed 
changes in the estimated-POM are indicative of changes in the 
percent variance attributed to overall modulations in the un- 
derlying rate under a wide range of assumptions. 

The excess variance attributed to rate modulations was also 
quantified using principal component analysis (PCA). The accumu- 
lated excess variance in the initial principal neurons, above the 
background noise level, was similar to the average POM. 
Furthermore, the variance of the 1 principal neuron, which 
represented the variance of the most correlated linear-component of 
the neural activity, was higher in brain control than in pole control. 
Thus, the PCA indicated that the neural activity during brain control 
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included a larger component of correlated activity compared to pole 
control, in support of the conclusions from the POM analysis. 

We also evaluated the percent variance of the binned spike- 
counts that can be attributed to velocity or kinematics modula- 
tions, and defined it as the PVM or PAM, respectively. Comparing 
the changes in POM with the changes in PVM or PKM revealed 
that the enhanced modulations cannot be attributed solely to 
velocity or even kinematics modulations. Furthermore, as training 
progressed, and the monkeys became more proficient in operating 
the pole and the BMI, the POM decreased while the PVM and 
PKM remained relatively constant. The observed trend in POM 
during BMI operation paralleled the effect on behavioral 
performance, which degraded during initial BMI operation and 
improved with subsequent training. 

The firing rate modulations were especially strong when the 
monkeys controlled the cursor without moving their arms, 
although the neural signals that would be related to the movement 
of the arm were irrelevant in that mode. The exact source of these 
extra firing modulations cannot be assessed using the current BMI 
experiments, since only the movement kinematics was measured 
directly. However, the results motivate some hypotheses, as 
detailed below, which would be explored in future investigations. 

The observed enhancement in neuronal rate-modulations may 
result from internal representation and processing of prediction- 
and execution-errors, which intensify when starting to operate the 
BMI and weaken gradually with subsequent training. Prediction- 
and execution-errors [29,30] are prevalent in novel behavioural 
contexts, possibly due to lack of appropriate kinematics or 
dynamic internal models [31-36]. ‘The motor system may correct 
the execution-errors on-line using feedback control [37,38] or 
initiate explorative activity to learn the new environment. Thus, 
the enhanced activity in different cortical areas may reflect the 
larger prediction and execution errors during initial BMI 
operation, and the resulting feedback and/or explorative activity. 

During the BMI experiments, the internal models are expected 
to adapt to improve BMI operation and capture its input/output 
relationship. This may explain the changes in the tuning curves of 
the recorded neurons [6,17], and the improvement in task 
performance with training [7]. Regardless of the nature of the 
adaptation mechanism, the improved task performance implies 
that execution errors diminished with training. Thus, our 
hypothesis is supported by the observed reduction in POM with 
training, despite the relatively invariant PVM. 

Our hypothesis can also explain the relatively moderate increase 
in POM when switching to BCWH, compared with the higher 
increase when switching to BOWOH. During BCWH, both the 
visual and proprioceptive feedbacks were relevant but provided 
conflicting error signals: the visual feedback indicated that the cursor 
was deviating from the desired trajectory, while the proprioceptive 
feedback indicated that the arm was following the desired trajectory. 
Under these conflicting error signals, the response would be 
attenuated and result in only moderately higher POM. During 
BCWORH, only the visual feedback was relevant and could trigger 
a full response, and thus an even higher POM. 

Different cortical areas, including the ones studied here, have 
been shown to be involved in the different computational aspects 
of sensorimotor adaptation. Studies of prism adaptation demon- 
strated that the dorsal premotor area (PMd) plays an important 
role in on-line error corrections [39,40,41]. Remapping, which 
facilitates adaptation, seems to occur in the posterior parietal (PP) 
cortex [39]. The primary motor area (M1) is involved in both 
motor performance and the acquisition of new motor skills 
[42,43,44]. The supplementary motor area (SMA) is strongly 
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involved in motor planning but is also involved in motor execution 
and seems to represent mainly the dynamics of the movement [45]. 
Our results are consistent with the reported role of the different 
cortical areas in on-line error-correction. In line with the role of 
PMd neurons in error correction, the increased execution error 
during initial BMI operation should increase the modulation of 
their activity. Indeed our results indicate that the POM of PMd 
neurons was significantly higher during brain control than during 
pole control (Figure 5 and Figure 6 right columns, Figure 7). Thus, 
the increased modulations of PMd neurons may reflect the 
corrective response of an internal feedback controller. Increased 
modulations in PMd are expected to result in increased 
modulations in M1, in agreement with the observed change in 
the POM of M1 neurons. Additionally, the increased modulations 
in SI neurons may result from proprioceptive prediction errors. 
Depending on the nature of the motor response, the enhanced 
neural modulations may encode only the magnitude or also the 
direction of the errors. Explorative activity might be generated based 
solely on the magnitude of the prediction or execution error. In 
contrast, feedback-corrections require also the representation of the 
direction of the error. In either case the enhanced modulations are 
expected to carry viable information during initial BMI operation 
that could be used for its improvement. Future experiments would be 
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directed at explormg whether and how the enhanced modulations 
are tuned to prediction and execution errors. 

Our hypothesis suggests that long-term use of a BMI would lead 
to its incorporation into internal models that would facilitate the 
accurate operation of the external actuator as if it was a natural 
limb. Exploring the nature of the enhanced modulations would 
facilitate the development of efficient BMIs and would provide 
further insight into the mechanism of adaptive skill acquisition. 
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The ability to walk may be critically impacted as the result of neurological injury or disease. 
While recent advances in brain—machine interfaces (BMIs) have demonstrated the feasibility 
of upperlimb neuroprostheses, BMIs have not been evaluated as a means to restore walking. 
Here, we demonstrate that chronic recordings from ensembles of cortical neurons can be 
used to predict the kinematics of bipedal walking in rhesus macaques — both offline and in real 
time. Linear decoders extracted 3D coordinates of leg joints and leg muscle electromyograms 
from the activity of hundreds of cortical neurons. As more complex patterns of walking were 
produced by varying the gait soeed and direction, larger neuronal populations were needed to 
accurately extract walking patterns. Extraction was further improved using a switching decoder 
which designated a submodel for each walking paradigm. We propose that BMIs may one day 
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INTRODUCTION 

Bipedal locomotion control is of great interest to the field of brain— 
machine interfaces (BMIs), i.e. devices that utilize neural activity 
to control limb prostheses (Chapin, 2004; Fetz, 2007; Lebedev 
and Nicolelis, 2006; Nicolelis, 2001; Schwartz et al., 2006; Taylor 
et al., 2002). Since locomotion deficits are commonly associated 
with spinal cord injury (Dietz, 2001; Dietz and Colombo, 2004; 
Rossignol et al., 2007; Scivoletto and Di Donna, 2008; Wood- 
Dauphinee et al., 2002) and neurodegenerative diseases (Boonstra 
et al., 2008; Green and Hurvitz, 2007; Morris, 2006; Pearson et al., 
2004; Sparrow and Tirosh, 2005; Yogev-Seligmann et al., 2008), 
there is a need to seek new potential therapies to restore gait control 
in such patients. While the feasibility of a BMI for upper limbs has 
been demonstrated in studies in monkeys (Carmena et al., 2003, 
2005; Serruya et al., 2002; Taylor et al., 2002; Velliste et al., 2008; 
Wessberg et al., 2000) and humans (Hochberg et al., 2006; Patil 
et al., 2004), it remains unknown whether BMIs could aid patients 
suffering from lower limb paralysis, e.g. by driving a leg prosthesis 
or artificial exoskeleton (Fleischer et al., 2006; Hesse et al., 2003; 
Veneman et al., 2007). 

Pioneered by Borelli (Borelli, 1680), investigations in biologi- 
cal systems have generated a wealth of knowledge about the bio- 
mechanics (Alexander, 2004; Andriacchi and Alexander, 2000; 
Dickinson et al., 2000; Koditschek et al., 2004; Ounpuu, 1994; 
Saibene and Minetti, 2003; Stevens, 2006; Vaughan, 2003; Zajac 
et al., 2002; Zatsiorky et al., 1994) and neurophysiological mecha- 
nisms underlying locomotion (Beloozerova et al., 2003; Deliagina 
et al., 2008; Drew et al., 2004; Georgopoulos and Grillner, 1989; 
Grillner, 2006; Grillner and Wallen, 2002; Grillner et al., 2008; 


allow severely paralyzed patients to walk again. 
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Hultborn and Nielsen, 2007; Kagan and Shik, 2004; Orlovsky et al., 
1999; Takakusaki, 2008). Many such biological principles are being 
applied to robotic locomotion (Azevedo et al., 2007; Kimura et al., 
2007; Morimoto et al., 2008; Nakanishi et al., 2004). 

A neuroprosthetic for the restoration of locomotion could be 
designed in several different ways. For example, a very simplified 
interface could be built, extracting only speed and directional 
information from neural activity, and offloading all precise move- 
ment control to onboard computerized systems. However, such a 
neuroprosthetic would not offer much more functionality to the 
user than a motorized wheelchair. Alternatively, a far more ambi- 
tious neuroprosthetic could attempt to extract control signals for 
every articulated joint in the robotic prosthetic, trusting the user 
to learn to control every aspect of the limb’s usage. However, much 
of balance and postural adjustments involve involuntary mecha- 
nisms (Deliagina and Orlovsky, 2002; Grillner et al., 2007; Maki 
and Mcllroy, 2007), and the lack of perfect decoding and sensory 
feedback could result in falls and injuries. Our approach takes the 
middle ground, if we can decode key walking parameters: step time, 
step length, foot location, and leg orientation, while offloading 
other automatic-level controls: foot orientation, load placement, 
balance, and safety concerns to onboard computerized systems, 
then we can achieve a BMI that follows the general commands 
of the user while enforcing stability, and overriding motions and 
configurations likely to result in falls. 

We have therefore extended our laboratory’s BMI approach 
(Carmena et al., 2003; Lebedev and Nicolelis, 2006; Wessberg 
et al., 2000) to investigate whether cortical activity can be utilized 
to extract the kinematics of bipedally walking rhesus macaques. 
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Although macaques are quadrupeds (Chatani, 2003; Courtine et al., 
2005a), they can be trained to walk bipedally (Hirasaki et al., 2004; 
Matsuyama et al., 2004; Mori et al., 2001, 2004; Tachibana et al., 
2003). In the present study, rhesus macaques walked bipedally on 
a treadmill while neuronal ensemble activity was recorded from 
the representation of the lower limbs in the primary motor (M1) 
and somatosensory (S1) cortices. We confirmed that a BMI using a 
series of independent linear decoders can accurately extract walking 
patterns from the activity of multiple cortical areas. As the loco- 
motion task demands increased, significantly more neurons were 
needed to achieve accurate extraction. Finally, we demonstrated 
that locomotor parameters can be extracted in real time to control 
artificial actuators that reproduce walking patterns. 


MATERIALS AND METHODS 

EXPERIMENTAL SETUP 

All studies were conducted with approved protocols from the Duke 
University Institutional Animal Care and Use Committee and were 
in accordance with the NIH guidelines for the Care and Use of 
Laboratory Animals. 

Two adult female rhesus macaques were trained to walk biped- 
ally on a custom modified treadmill. The treadmill was a human 
fitness treadmill, modified to be hydraulically driven so that the 
pump motor could be located remotely, in a different room than 
the experimental setup and thus reducing the electronic noise that 
could be picked up by the neuronal recording equipment. Treadmill 
speed and direction were separately controlled via independent 
throttle and flow routing controls. 

Around the treadmill was a metal frame that supported both the 
recording equipment and the monkey restraints. The monkeys were 
loosely restrained by an adjustable 5-degree of freedom neckplate. 
By adjusting the three dimensional position of the neckplate as well 
as two dimensional tilt, the monkeys could be comfortable in their 
normal walking posture. Additionally, a bar was placed within reach 
of the monkeys’ arms to allow for a comfortable stability aid. Each 
of the monkey’s legs and arms were unrestrained during experi- 
mental walking sessions. After 1 month of training, the monkeys 
learned to walk bipedally on the treadmill (Figure 1A). Monkey 1 
was trained to walk in both the forward and backward direction at 
varying speeds (Figure 1C), and Monkey 2 was trained to walk only 
in the forward direction. The treadmill speed varied in the range 
—0.5 to 0.5 m/s. The monkeys received food treats during walking 
sessions lasting 40-60 min, which encouraged them to face to the 
front of the treadmill. 


LIMB MOVEMENT TRACKING 

During experimental sessions, movements of the right legs of the 
monkeys were tracked using a wireless, video-based tracking sys- 
tem. The three dimensional coordinates of fluorescent markers 
applied to the hip, knee, and ankle were tracked using two cameras 
at a 30 Hz frame rate. Initially a commercial offline system (SIMI 
Motion 3D) was used for tracking purposes; in later sessions a 
custom real-time video-based tracking system developed in the lab 
was used (Peikon et al., 2007). Data were cross-validated between 
the two systems to ensure that both produced equally accurate 
results and to ensure no biases were present in our custom system. 
To ensure consistent placement of the markers which were tracked 


on video, each of the monkeys was tattooed over the hip, knee, and 
ankle joints of their right legs. The markers themselves were applied 
before each session, and were made of fluorescent, non-toxic stage 
makeup. This approach offered several advantages. First, the mark- 
ers were virtually weightless on the skin of the monkeys and thus 
did not impact the walking mechanics or cause distraction. Second, 
using fluorescent markers allowed us to achieve very high contrast 
ratios between markers and background on the recorded video 
stream. By turning visible light down to a low level, lowering the 
camera aperture, and bathing the experimental setup with a safe, 
filtered UV (or black) spotlight, the only significant source of visible 
light picked up by the cameras came from the markers which were 
being tracked. While the current tracking system is only capable of 
tracking the right leg, future versions will track both legs simulta- 
neously, allowing us to investigate bipedal predictions. Figure 1B 
illustrates frames of captured video that show an example of the 
bipedal walk cycle for each monkey. These figures have increased 
levels of ambient lighting to aid in viewing the step cycle. Both mon- 
keys walked in a stereotyped bipedal manner and slightly leaned 
forward for comfort with arms holding a bar for support. 


KINEMATIC ANALYSIS 

The monkeys’ limb tracking information was used to extract a 
number of experimentally relevant parameters in addition to the 
X, Y and Z coordinates of the joint markers. We extracted joint 
angles (hip and knee), foot contact with the treadmill, walking 
speed, step frequency and step length. All these parameters were 
calculated from the joint position data provided that the markers 
were not occluded. The episodes during which any of the markers 
was occluded (typically, when the monkeys turned their bodies) 
were excluded from the analyses. Our video analysis algorithm used 
a combination of mathematical techniques to calculate additional 
parameters. Treadmill speed was extracted from the video tracking 
data. The frequency of the step was extracted from the kinematic 
movement pattern of the ankle in the axis of treadmill motion. 
Furthermore, a combination of ankle displacement during the 
step phase and treadmill speed was used to extract stride length 
with respect to the moving surface of the treadmill. Foot contact 
was extracted first from a filtered height (Y-axis) threshold, which 
determined when the ankle joint was flat on the surface of the 
treadmill. However, a simple height threshold was not adequate to 
reject shuffling walking types, and a further check was instituted, 
which rejected periods without constant X-velocity of the ankle 
equal to treadmill speed. 


SURGICAL AND ELECTROPHYSIOLOGICAL PROCEDURES 

During the implantation surgeries, each of the two adult female 
rhesus macaques was anesthetized and placed in a stereotaxic appa- 
ratus (for a full description see Nicolelis et al. 2003). All surgical 
procedures conformed to the National Research Council’s Guide for 
the Care and Use of Laboratory Animals (1996) and were approved 
by the Institutional Animal Care and Use Committee. 

A series of small craniotomies were made, both to grant access 
to the brain for the microwire arrays and for anchoring the den- 
tal acrylic to the skull. In each animal, multiple microwire arrays 
were chronically implanted in several cortical areas (Figure 2). The 
implanted areas were chosen based on previous cortical mapping 
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FIGURE 1 | Experimental setup and Kinematic Analysis. (A) Diagram of an average over an epoch of constant treadmill speed during variable speed 
the bipedal walking setup, consisting of a custom modified, hydraulically driven and direction sessions. (E) The relationship between step length and walking 
treadmill; wireless, 2-camera tracking of kinematics; and a MAP neural speed. As in (C), blue and red correspond to the actual and predicted values, 
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studies (Nicolelis et al., 2003; Tanji and Wise, 1981; Wise and Tanji, 
1981a,b). 

Accordingly, multi-electrode arrays were inserted medially in the 
cortex approximately 2 mm in front of the central sulcus to target 
M1 and 2 mm behind it to target S1 (Figure 2). The arrays were 


matched in size to the cortical representations of the hind limb. 
Monkey 1 was implanted with 50 um tungsten chronic microwire 
arrays in the leg representation of both the primary somatosensory 
(S1) and primary motor (M1) cortices. One 8 by 8 square array with 
electrodes spaced 1 mm apart (inserted 2.5 mm deep in the cortex) 
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FIGURE 2 | Diagram of cortical implants. A generic outline of cortical sulci 
(view from the top) is shown to illustrate the locations of the implants. Positive X 
coordinates correspond to rostral brain areas, while negative X coordinates 
correspond to caudal brain areas. (A) Implanted locations in Monkey 1. Both 
hemispheres were implanted. Two 32-electrode arrays were implanted in the left 
M1, one 32-electrode array in the left S1 and one 64-electrode array in the right 
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M1. (B) Implanted locations in Monkey 2. The left hemisphere was implanted 
with two 64-electrode arrays, one placed in M11 and one in S1. (C-F) 
Configuration of the microelectrode arrays. The electrode shafts were arranged 
into 1-mm spaced grids. In multi-layered configurations, each shaft consisted of 
two (C,E) or four (D) microwires of different length. The difference in depth 
between the electrode layers was 0.3 mm. 


was implanted rostral to the central sulcus. A 6 by 6 double-layered 
square array, using the same electrode spacing, was placed caudal 
to the central sulcus, 1.5 mm deep. In the double layer implant, 
the second layer of electrodes was 300—400 tm shallower than the 
first layer. Monkey 2 was also implanted with chronic microwire 
arrays in the leg representation of both the primary somatosensory 
(S1) and primary motor (M1) cortices. In M1, two 3 by 6 double 
layer arrays were implanted, while in S1 a single double layer 3 by 
6 array was implanted. In all implants one layer was 300 pm deeper 
than the other layer. All electrodes in Monkey 2 were stainless steel 
microwires ranging in diameter from 40 to 60 um. In both monkeys 
the placement of the electrodes was accomplished using stereotaxic 
coordinates and connectors for the arrays were embedded in a head 
cap made of dental acrylic. 


Upon recovery from the surgical procedure, the receptive fields 
of individual S1 neurons and multi-unit activity were briefly exam- 
ined in the awake monkeys by lightly touching and palpating their 
hind limbs. This examination confirmed that in both monkeys the 
implanted S1 sites represented the thigh, calf and the foot. No clear 
somatosensory responses were identified for the M1 implants. In 
Monkey 2, we briefly tested motor responses to cortical micros- 
timulation under ketamine anesthesia. The microstimulation of 
M1 evoked hind limb movements due to proximal muscle con- 
traction in agreement with Hatanaka et al. (2001) and Tanji and 
Wise (1981). 

A total of 200-300 well sorted single units were recorded from 
implants in both monkeys per experimental session. After the mon- 
keys were placed in the treadmill-mounted restraint, head stage 
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amplifiers were attached to the head-cap connectors. A flexible 
wire harness, in turn, connected the headstages to a 128 channel 
Multichannel Acquisition Processor, or MAP, (Plexon, Inc., TX, USA) 
recording system. Neuronal units were sorted in real time using the 
templates defined in MAP software. The ratio of the amplitude of 
each sorted unit to the amplitude of electrical noise was, on average 
3.34 + 1.66 (mean + standard error). The quality of online sorted 
single units was further examined by analyzing the refractory period, 
estimated from the interspike intervals. For each unit to be qualified 
as single unit, in addition to having a distinct shape and amplitude 
(Nicolelis et al., 2003), it had to exhibit a refractory period greater 
than 1.6 ms (Hatsopoulos et al., 2004). Using these criteria, 66.0% 
of the recorded units were single units, and 34.0% were classified as 
multi-unit neuronal activity. Overall, extraction of locomotion pat- 
terns from single units versus multi-units yielded similar results. 

Electrophysiological recordings spanned 399 days in Monkey 1 
and 56 days in Monkey 2. The implants were connected to a mul- 
tichannel recording system (Plexon, Inc., TX, USA) using light flex- 
ible cables. The total number of simultaneously recorded units 
ranged from 180 to 238 in Monkey 1, depending on the record- 
ing day, and from 173 to 334 units in Monkey 2. In Monkey 1, 
we recorded from 111.2 + 19.3 units (mean + standard deviation; 
standard deviation reflects day to day variability) in left M1, from 
38.1 + 6.1 units in left S1 and from 55.6 + 16.0 units in right M1. In 
Monkey 2, we recorded from 106.0 + 19.9 units in left M1 and from 
166.0 + 31.0 units in left $1. Statistical analysis (Wilcoxon signed 
rank test) confirmed that average neuronal firing rates increased 
during walking in both M1 (P < 0.001) and S1 (P< 0.001) in each 
monkey. The average firing rate of M1 units was 7.5 + 8.9 spikes/ 
s during standing versus 15.0 + 13.4 spikes/s during walking in 
Monkey 1, and 7.7 + 8.7 spikes/s versus 11.4+11.9 spikes/s in 
Monkey 2. In S1, the average rates were 8.7 + 7.7 spikes/s during 
standing versus 16.6 + 10.3 spikes/s during walking in Monkey 1, 
and 14.7 + 12.9 spikes/s versus 24.3 + 20.3 spikes/s in Monkey 2. 

The MAP system was used for receptive field testing and for obtain- 
ing neuronal recordings during walking sessions. Electromyogram 
(EMG) signals were recorded from Monkey 1’s shaved skin surface, 
centered over the soleus, rectus femoris, and tibialis anterior muscles 
of both legs. Gold disc electrodes (Grass Instrument Co., RI, USA) 
were placed over conductive gel to obtain the EMG recordings. These 
EMGs were amplified up 10,000 times, band-pass filtered between 
100 Hzand 1 kHz, rectified, and recorded using the MAP recording 
system to ensure consistent timing. 


MODELS UTILIZED FOR PREDICTING LEG KINEMATICS 

All the leg kinematic parameters extracted were reconstructed 
from neuronal ensemble activity using the linear decoding algo- 
rithm called the Wiener filter (Carmena et al., 2003; Haykin, 2002; 
Lebedev et al., 2005; Wessberg et al., 2000). The Wiener filter rep- 
resented each decoded parameter as a weighted sum of neuronal 
rates measured before the time of decoding. 


N 10 


X(t)=b+ YY w,nlt -(j-Dt]+ ef) (1) 


i=l j=1 


where X(t) is the value of the decoded parameter (for exam- 
ple, X-coordinate of the ankle marker) at time t, n, is the firing 


rate of neuron i, N is the total number of neurons, (j — 1)t is tap 
delay for tap j, w,, is the weight for neuron i and tap j, b is the 
y-intercept, and ¢(t) is the residual error. For extracting marker 
coordinates, joint angles, and foot contact with the treadmill the 
tap length parameter t was set to 50 ms, and the number of taps 
(time bins of neuronal data) was set to 10, that is neuronal rates 
were sampled in a 500 ms window preceding the time of decoding. 
For extracting slower modulated characteristics such as walking 
speed and step length, a 5 s sample window was used composed 
of ten 500 ms taps. 

Prediction of leg kinematics was performed using multiple 
Wiener filters applied to the activity of the entire population of 
the recorded neurons or subpopulations recorded in separate cor- 
tical areas. Thus, the activity of simultaneously recorded cortical 
neural ensembles allowed us to simultaneously extract a variety of 
motor parameters: position of hip, knee, and ankle; hip and knee 
angles; as well as foot contact, direction of walking, and periods 
of standing still. We used multiple linear models (Wiener filters; 
Carmena et al., 2003; Haykin, 2002; Lebedev et al., 2005; Wessberg 
et al., 2000) to describe the relationship between these parameters 
and neuronal ensemble activity. To calculate model weights, first 
Eq. 1 was converted to matrix form as: 


X=NWte (2) 


where X is the matrix of actual parameters, N is the matrix of neu- 
ronal rates, W is the weights of the model, ¢ is the error. Each row 
of N corresponds to a specific time and each column is a vector of 
data for a particular neuron and time lag. Since our models took 
into account ten lags, matrix N had ten columns for every neuron. 
The y-intercept was calculated from a column of ones prepended 
to matrix N. We then solved for matrix W by the following: 


We inv(N’N)N*X (3) 


Each Wiener filter was first trained (i.e. the values of weights 
W were calculated) and then used as the decoder for new data. 
Accordingly, each experimental record (10-15 min) was split in 
two halves: the training data and the predicting data. The model 
was trained on the first half of the experimental data and pre- 
dictions were obtained using the second half. Decoding was also 
conducted for the reverse arrangement: training the models on 
the second half and using it to predict the first half. In addition to 
these offline analyses conducted for 80 experimental records (66 
with Monkey 1, 14 with Monkey 2), real-time extraction was per- 
formed in 22 experimental sessions with Monkey 1. For real-time 
extraction, the neuronal and kinematic data were first recorded for 
5 min while multiple Wiener filters were trained, and then online 
extraction of walking parameters was performed for 5—10 min. 


GENERAL AND SWITCHING MODEL 

To test whether models trained to accurately predict motor param- 
eters for a specific behavioral paradigm would retain general kin- 
ematic prediction accuracy during alternative behaviors (i.e. be 
able to generalize to new paradigms), we trained models for one 
direction of movement (e.g. forward walking) and tested them 
in the other (e.g. backward walking; for an example of this, see 
Figure 7). This allowed us to ask whether a model that is trained 
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to predict leg kinematics while the monkey walks forward would 
retain prediction accuracy when applied to sessions of backward 
walking and vice versa. Next, we investigated whether our models 
would benefit from training on a variety of behavioral data, in this 
case both forward and backward walking. To address these ques- 
tions, a series of linear models were trained on selected behavioral 
epochs. The models were trained on: (1) only forward walking, 
(2) only backward walking, and (3) an equal mixture of the two 
walking paradigms. Each model was trained on an equal length 
of time to avoid any performance biases and then tested on its 
ability to accurately predict walking forward or walking backward 
separately. 

The switching mode was used to handle the conditions in which 
the monkey’s locomotion consisted of two different paradigms: 
alternating periods of walking forward or walking backwards. 
Separate submodels were trained to decode each of these walking 
paradigms, and the paradigm predictor model served to detect 
the walking paradigm and select the appropriate submodel. The 
brief periods during treadmill mode switching when the monkey 
was standing still were classified as forward walking rather than 
introducing a third behavioral category. In our implementation, 
the switching model was a combination of three linear decoding 
models: a model for predicting forward walking, a model for pre- 
dicting backwards walking and the paradigm predictor model (the 
switch). These models were arranged in a two layer structure (see 
Figure 7F) with the paradigm predictor model controlling a toggle 
between the two kinematic submodels which were then shunted to 
the final output of the switching model. Thus, when it was deter- 
mined that the monkey was walking forward, one submodel was 
used to produce the output, and when backwards walking was 
detected, the other submodel was used. 

To avoid any bias in our comparisons between the generalized 
and switching model, they were both trained on the same amount 
of data. This means that while the single generalized model was 
trained over the full training window, each of the submodels of the 
switching model were only trained on the portions of the data when 
the monkey’s behavior fell in the relevant behavioral paradigm. 
Only the classifier portion of the switching model was trained on 
the full training window. This way, a true comparison of perform- 
ance was achieved, and we could test whether the disadvantages of 
training the kinematic model on less total data were overcome by 
the advantages in prediction accuracy that come from behavioral 
segregation. 


MODEL PERFORMANCE METRICS 

The signal to noise ratio (SNR) used in signal processing is a ratio 
between the power of the signal and the power of the noise, that 
is, the ratio of the squared amplitude of the signal and the squared 
amplitude of the noise. For our purposes, the signal was defined as 
the actual variable that we predicted. We calculated the variance, or 
power, of the signal by subtracting out the mean of the signal, then 
squaring and averaging the amplitude above or below that mean. 
The noise was the difference, or error, between the extracted and 
the actual signal. The error was calculated by subtracting the actual 
parameter from the extracted parameter, squaring the differences, 
then averaging to get the mean squared error, or the power of the 
noise. The ratio between these two was the SNR [or the signal 


to error ratio (SER)]. We then converted the ratio into a decibel 
(dB) scale: 


(4) 


SNR(X,X)=10log,, = 
MSE(X) 


Var(X) ] 


Where X is the actual parameter, X is the prediction of that 
parameter, Var is variance, and MSE is mean squared error. 
Theoretically, SNR can change from minus infinity (extremely poor 
predictions) to infinity (extremely good predictions). A SNR of 0 
meant the signal and noise were in equal proportion in our pre- 
dicted output. A SNR>0 meant we were extracting more signal than 
noise, and an SNR <0 meant there was more noise than signal, but 
the signal was still present for negative SNRs close to 0. SNRs equal 
to 10, 20, and 30 correspond to the ratio between the variances of 
10, 100 and 1000, respectively 

Additionally, Pearson’s correlation coefficient, R, between the 
known signal and the predicted output was calculated: 


a cov(X os ) 
R(X,X)=——+ (5) 
0,0; 

Where X is the actual parameter, X is the prediction of that 
parameter, cov is covariance, and Oo, ando 3 are the standard devi- 
ation of X and X respectively. R can change in the range —1 to 
1, with 1 corresponding to the highest possible correspondence 
between the actual and extracted values, and 0 corresponding to 
the absence of correspondence. Negative values of R, correspond- 
ing to an inverse relationship between the actual and extracted 
characteristics, occasionally occur when the extraction is poor, and 
R fluctuates around 0. 

Signal to noise ratio proved to be a more sensitive measure com- 
pared to R in these analyses. This was because R describes the corre- 
spondence of signal waveforms, but is insensitive to amplitude scaling 
and offsets. SNR is sensitive to errors introduced by these factors, 
which is important for the purposes of efficient BMIs that require 
that the output of the model matches the true signal in all aspects. 
Additionally, R often saturates quickly for large ensemble sizes, 
whereas SNR better tracks the dependency of model performance 
on the ensemble size. A metric similar to SNR, called SER was used 
in previous studies in which behavioral parameters were extracted 
from neuronal data (Kim et al., 2006; Sanchez et al., 2002). 


NEURON DROPPING ANALYSIS 

Neuron dropping analysis (NDA) is a conventional way to charac- 
terize the prediction performance of neuronal ensembles of differ- 
ent size (Carmenaet al., 2003; Lebedev et al., 2005, 2008; Wessberg 
et al., 2000). In this analysis, a number of decoding models were 
trained on random subsets of sorted neurons, ranging in size from 
a single neuron to the entire bank of sorted neurons. To charac- 
terize the performance of these decoding models, we calculated 
neuron dropping curves which described decoding accuracy as 
a function of neuronal sample size (Wessberg et al., 2000). The 
dropping curves were calculated by pooling randomly selected 
subsets of neurons and running the decoding model only for them. 
At each population subset size, five random subsets of neurons 
were generated and used to train separate predictive models. The 
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predictive strength (SNR) of these models was calculated and plot- 
ted as scatter plots (for examples of this, see Figures 6A—F and 
7D,F). These scatter plots were then fitted using a power curve 
with simple linear decay: 


P 
Y = Vortset + Vamplitude (x = Hise) + MgecayX (6) 


where x is neuronal subset size and y is SNR. Five parameters 
were fitted to generate the curves: y_,,., translated the entire curve 
up or down, Rae translated left or right, Y eciiosis defined the height 
of the fitted curve, p was the power of the curve and described the 
rate of the climb in SNR with increases in neuronal subset size, and 
M4gcqy described the slope of the linear decay. 

For our analysis examining predictions of several kinematic 
variables simultaneously from the same neuronal population, we 
used a modified version of the NDA. Random subpopulations of 
the neuronal ensemble were selected and used to train models 
for a random combination of kinematic parameters simultane- 
ously. Simultaneous prediction accuracies were calculated to be 
the minimum SNR level reached by all the predicted variables at 
each neuronal ensemble size. The overall simultaneous prediction 
dropping curves were then fitted by power curves (Eq. 6). The 
curves were normalized and averaged across all combinations of 
kinematic variables. The final curves were thresholded at several 
levels of prediction accuracy (0.75, 0.85 and 0.95) to determine the 
minimum number of neurons needed to predict each number of 
kinematic variables simultaneously with the given level of accuracy 
(for an example of this, see Figures 8A,B). Essentially, we calculated 
the expected number of neurons that would be required to predict, 
to a high accuracy threshold, any number of kinematic variables 
simultaneously in our experimental setup. 


REAL-TIME SYSTEM 

We also developed a real-time BMI software suite capable of run- 
ning all aspects of our experimental setups, including visual display, 
behavioral and multi-electrode neural recordings, model calcula- 
tion and real-time predictions of kinematic and dynamic param- 
eters. Written in Microsoft Visual C++, this BMI software suite 
can input behavioral data, multichannel extracellular single and 
multi-unit recordings, EMGs, and local field potentials. 

Using a graphical user interface, models were trained to gen- 
erate real-time predicted kinematics or EMGs from neural data. 
While our software suite currently implements the Wiener Filter, 
the Kalman Filter with optional principle components analysis 
dimensionality reduction, the n-th order Kalman Filter, and the 
n-th order Unscented Kalman Filter, in the real-time experiments 
described in this study we have used only the Wiener Filter option 
for extracting kinematic, dynamic, and EMG data. 

The outputs of multiple real-time predictive models were dis- 
played on a computer screen and simultaneously streamed over 
the network using a User Datagram Protocol. 


RESULTS 

KINEMATICS OF BIPEDAL WALKING 

During walking, both monkeys adopted a posture in which they 
leaned slightly forward while holding a bar and episodically 
assumed a more upright posture (Figures 1A,B). The right leg 


was video-tracked in real time using fluorescent markers applied 
to the hip, knee and ankle joints. The monkeys’ walking kinematics 
exhibited many basic features previously reported in both humans 
(Capaday, 2002; Grieve and Gear, 1966; Hirokawa, 1989; Murray 
et al., 1964, 1970; Nielsen, 2003; Perry, 1992; Vaughan et al., 1992) 
and monkeys (Hirasaki et al., 2004; Matsuyama et al., 2004; Mori 
et al., 2001; Tachibana et al., 2003). Step frequency rapidly increased 
as the treadmill speed increased from 0.05 to 0.2 m/s and began to 
plateau above 0.2 m/s (blue traces in Figure 1D). Step frequency 
during backward walking plateaued at a slightly higher value (1.1 
steps/s) versus forward walking (1.0 steps/s). Step length also 
increased as the treadmill speed increased with a slight tapering 
off at higher speed (blue traces in Figure 1E). The stance phase 
constituted approximately 65% of leg stepping cycle (Monkey 1: 
69.6 + 13.1% during forward walking, 66.7 + 16.2% during back- 
ward; Monkey 2: 57.6 + 9.7% during forward). 

During forward walking, the ankle and knee moved backward 
at a nearly constant speed after the foot touched the treadmill 
(Figure 3A). At approximately mid-stance, the ankle started 
to lift, followed by the knee (Figure 3B). These movements 
were accompanied by a lateral displacement (Z-dimension) of 
the ankle because of foot rotation (Figure 3C). The knee angle 
monotonically increased during the stance phase as the knee joint 
extended during backward movement of the leg, and knee flexion 
started shortly before foot liftoff and continued during the swing 
(Figure 3D). The soleus and rectus femoris were active during 
the stance phase and relaxed during the swing (Figure 3E). The 
tibialis anterior was activated in a reciprocal manner: active dur- 
ing swing, relaxed during stance. This pattern of muscle activity 
is similar to EMG patterns during human walking (Capaday and 
Stein, 1986; Ivanenko et al., 2008; Kadaba et al., 1985; Shiavi et al., 
1987; Winter and Yack, 1987). Examples of video frame during 
stance and swing are shown in Figure 3F 

The discharge rate for each of the cortical neurons peaked at a 
preferred phase of the step cycle (Figure 4). The pattern of corti- 
cal activity during forward walking (Figure 4A) was different for 
each recorded cortical area. In contralateral (left) M1, the number 
of neurons whose rates peaked during the stance was approxi- 
mately equal to the number of neurons with the peak rate during 
the swing (61 + 11 versus 63 + 11, respectively, P > 0.05 Student’s 
t-test). However, since the swing was shorter than the stance, the 
density of peak rates per unit of the step cycle (the slope of the 
red line in Figure 4A, lst row) was lower for the stance phase 
(87.2 + 18.6 peaks/s during stance versus 261.0 + 29.0 peaks/s 
during swing, P< 0.001 Student’s t-test), especially for its initial 
phase preceding the stance to swing transition. Ipsilateral (right) 
M1, showed a different pattern of neural activity in this analysis 
which used the movements of the right leg to designate the step 
cycle, with more neurons peaking during the stance phase than the 
swing phase (38 + 6 versus 20 + 6, respectively, P < 0.001 Student’s 
t-test; Figure 4A, second row), evidently because neural modula- 
tions there represented the out of phase movements of the left leg. 
Activity in S1 neurons was heavily concentrated at the swing phase, 
especially during the stance to swing transition (55.0% of S1 neu- 
rons had their peak activity in the 20% of the step cycle centered 
at the stance to swing transition; Figure 4A, third row), when the 
foot was lifted from the treadmill surface. 
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FIGURE 3 | Average changes in walking parameters during the step cycle. 
The data from all experimental sessions during which the monkey walked in 
forward direction with EMGs recorded were averaged. The average portions of 
the stance and swing phases of the step cycle are shown below each plot. The 
hip movements are not illustrated because they were small. Solid lines show the 
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actual values, and dashed lines show the values extracted from neuronal activity 
using the Wiener filter. Note the correspondence between the actual and 
extracted values. (A-C) X, Y, and Z position. (D) Knee angle. (E) EMG activity in 
the muscles of the right leg. (F) Examples of leg orientations classified as swing 
and stance. 


A different pattern of ensemble modulations was revealed during 
backward walking (Figures 4B,C). Notably, firing of the neurons 
tended to be more evenly spaced over the step cycle, especially in 
M1 (176.4 + 29.4 peaks/s during stance versus 196.0 + 19.6 peaks/s 
during swing, P > 0.05 Student’s t-test). When the cortical activity 
during backwards walking was sorted according to where the cells 
were most active in forward walking (compare Figures 4A,C), it 
was apparent that there was little correspondence between indi- 
vidual cells’ activity peaks between forward and backward walking. 
However, there was still a slight tendency for the neurons that were 
active in the swing phase during forward walking to remain active 
during the swing during backward walking (44.4% of the neurons 
with peak rate during the swing) and the same tendency for the 
stance-phase neurons (80.1%). This tendency was statistically sig- 
nificantly greater (P < 0.005 Student’s t-test) than the percentage 
of neurons expected to stay in the same phase (29.9% and 70.1%) 
if there was no correlation between the neurons’ peak firing phases 
in forward versus backward walking. Thus, although leg kinematics 
were similar during forward and backward walking, the underlying 
neuronal patterns were substantially different. 


EXTRACTING CHARACTERISTICS OF WALKING FROM CORTICAL 
ENSEMBLE ACTIVITY 

Multiple locomotion parameters were extracted using linear decod- 
ers which expressed the parameters as weighted sums of the neuro- 
nal firing rates (Carmena et al., 2003; Haykin, 2002; Lebedev et al., 
2005; Wessberg et al., 2000). We decoded X (horizontal), Y (verti- 
cal), and Z (lateral) Cartesian coordinates of the leg joints, state of 


foot contact with the treadmill, step length and frequency, walking 
speed, and the EMGs of multiple leg muscles. Figures 4 and 5E 
represent modulations of M1 and S1 discharges as “waves” of neu- 
ronal activity that occurred during each step. While the individual 
neuronal firing rates were highly variable from step to step at the 
millisecond scale (Figure 5E), combining the activity of many neu- 
rons in 100 ms bins using the linear decoder produced accurate pre- 
dictions of leg movements, as evident from average traces for joint 
coordinates (Figures 3A-C), joint angle (Figure 3D) and EMGs 
of leg muscles (Figure 3E), as well as step frequency (Figure 1D) 
and step length (Figures 1E,F) derived from predictions of joint 
positions. Figures 5A-E illustrates a 5 s epoch of forward walking 
during which 18 motor parameters were simultaneously extracted: 
joint coordinates (Figure 5A), joint angles (Figure 5B), state of 
foot contact (Figure 5C) and EMGs (Figure 5D). Figure 5F depicts 
direct extraction of step length and walking speed (characteristics 
modulated at a slower pace) from a different 50 s epoch. 

The average extraction performance is summarized in Table 1. 
Overall, accuracy in predicting motor parameters was in line with 
that previously obtained for extracting arm movements from M1 
activity (Carmena et al., 2003; Lebedev et al., 2005; Wessberg et al., 
2000). We observed SNR in the range —2 to 7 dB and correlation 
coefficients, R, in the range 0.2—0.9. In both monkeys, the best 
extracted parameters were those related to the X and Y coordinates 
of the ankle, and knee angle (SNR in the range 3.8—6.0 dB and Rin 
the range 0.79-0.87). The Z coordinate (lateral movements) was 
not predicted as well, because of minimal lateral movements of the 
joints during walking sessions. Likewise, the small amount of hip 
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FIGURE 4 | Neural activity in each cortical area during the step cycle. The 
neural data from experimental sessions during which the monkey walked in the 
forward and backward direction were separately averaged for many step cycles. 
Each horizontal line corresponds to the firing rate of a single neuron normalized 
by standard deviation (i.e. zscore). Each cortical area was analyzed separately 
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activity. The average durations of the stance and swing phases of the step cycle 
are shown below each plot. Swing was slightly longer during backwards 
walking. (A) Average neural activity during forward walking, ordered via peak 
activity during forward walking. (B) Average neural activity during backward 
walking, ordered via peak activity during backward walking. (C) Average neural 
activity during backward walking, ordered via peak activity during forward 
walking [i.e. neurons are ordered the same way in (A) and (C)]. 


movement produced during walking may have contributed to low 
prediction accuracy for the hip’s Cartesian coordinates compared 
to those of the knee and ankle. The X and Y coordinates of the knee 
and the hip were extracted with an average SNR in the range —0.7 
to 4.3 dB and average R around 0.42-0.79. The average SNR and 
R for hip angle ranged from 0.9-3.0 and 0.58—0.73, respectively. 
Meanwhile, extraction accuracy for foot contact state (swing or 
stance) ranged from 1.2—3.1 in SNR and 0.58-—0.61 in R, whereas the 
average SNR and R for the slowly modulated parameters were in the 
range —2.0 to 1.4 and 0.24—0.42 for walking speed, —1.8 to 0.9 and 
0.48-0.57 for step frequency and —1.5 to 1.9 and 0.30-0.40 for step 
length. The low average accuracy of predicting the walking speed, 
especially in Monkey 2, reflected the fact that in many experiments 
the treadmill speed was constant for long periods of time and thus 
had very low variance. Predictions of EMGs (Figures 3E and 5D) 
were better (P < 0.001, Student’s t-test) for the musculature of the 
right leg (SNR of 1.55 + 0.39) than for the left leg (0.76 + 0.17), 
likely because more neurons were recorded in the left hemisphere. 


Indeed, when prediction performance of equal size samples of neu- 
rons drawn from left versus right M1 was compared, contralateral 
M1 outperformed (P< 0.001, Student’s t-test) ipsilateral M1 in 
predicting either leg EMG. 


NEURON DROPPING ANALYSIS 

Figures 6A—F depicts NDA for the extraction of walking param- 
eters. Each dot in the scatter plots represents prediction accuracy 
(quantified as SNR) calculated for a randomly selected neuronal 
subset. The plots are fitted with exponential curves. NDA revealed 
that the prediction accuracy of joint coordinates monotonically 
increased with neuronal sample size (Figures 6A—C). In the par- 
ticular case of predicting foot contact, NDA revealed a pattern 
consistent with the known phenomenon of overfitting (Babyak, 
2004; Santucci et al., 2005), i.e. the SNR increased as the sample 
grew to 50 neurons, and it decreased thereafter (Figure 6D). In 
agreement with Table 1, the Y-coordinates (vertical movements) 
for the ankle (Figure 6A) and knee joints (Figure 6B) were the most 
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FIGURE 5 | Extraction of multiple characteristics of walking. In all traces, 
blue shows actual and red shows extracted value. All variables are shown over 
the same 5 s window, except in (F), where slow changing variables are shown 
over a longer 50 s window. (A) Three dimensional position variables. The X axis is 
in the direction of motion of the treadmill: the Y axis is off the surface of the 
treadmill (axis of gravity); and the Z axis is lateral to the surface of the treadmill, 
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but orthogonal to the direction of motion. (B) Joint angle variables. Hip angle is 
calculated assuming a fixed torso. (C) Foot contact, a binary variable defining the 
swing versus stance phase. (D) EMG variables. (E) Neuronal activity of 220 
single units sorted via cortical area and by phase within the step cycle. Firing 
rate is normalized for every unit. (F) Slow changing state variables (walking 
speed, step frequency and step length) plotted over a 50 s time window. 


accurately predicted of all Cartesian coordinates. X was the second 
best extracted coordinate. Prediction accuracy for the knee and hip 
angles was comparable to the prediction of X and Y coordinates 
for these joints (Figures 6B,C). 

Neuron dropping analysis for predictions of the ankle X coor- 
dinate was also performed separately for different cortical areas 
(Figures 6E,F). In both monkeys, walking parameters could be 
predicted using neuronal activity recorded in either M1 or S1 con- 
tralateral to the right leg, as well as ipsilateral M1 in Monkey 1, but 
none of these areas taken alone reached the prediction performance 
level of the entire population. Predictions using random popula- 
tions drawn from each individual cortical area were statistically 
significantly worse than predictions from equivalently sized ran- 
dom populations drawn from all cortical areas (Student’s t-test, 
P<0.001). For each area, the prediction performance improved 
with increases in population size. Slight overfitting was observed 
for S1 in Monkey 2 for ensemble sizes greater than 80 neurons 
(Figure 6F, blue dots and curve). In Monkey 1, predictions from 
contralateral M1 neurons (Figure 6E, red) were better than the 
predictions from ipsilateral M1 neurons (Figure 6E, green) or con- 


—= 


tralateral S1 neurons (Figure 6E, blue). In Monkey 2, predictions 
from S1 neurons were better than the predictions from M1 neurons 
(Figure 6F). 

The performance of the extraction algorithm, using M1 ver- 
sus S1 neurons, was further examined using a timing analysis in 
which a single 50 ms time window was used to measure extraction 
accuracy at various lags with respect to the present time (time 
0; Figures 6G,H). Peaks in prediction accuracy for the lags pre- 
ceding the present time (negative lags) indicated that neuronal 
modulations were predictive of future movements, suggesting a 
causal link between neuronal modulations and movements. Such 
a relationship was expected for M1 neurons and confirmed for 
both Monkey 1 (Figure 6G) and Monkey 2 (Figure 6H). Peaks 
for lags succeeding the present (positive lags) indicated a reverse 
relationship: present movements were the cause of future neuronal 
modulations. This was expected for S1 neurons, and confirmed 
in Monkey 2; the peak S1 extraction occurred at a positive lag 
(Figure 6H). S1 activity was generally not strongly predictive 
at any lag in Monkey 1. There were minor peaks in S1 predic- 
tive power for both positive and negative lags (Figure 6G). Still, 
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Table 1 | Average prediction accuracy. Correlation coefficient and signal to noise ratio (dB) for the predictions of different walking parameters. The numbers 


represent mean + standard deviation 


Monkey 1 Monkey 2 
R SNR (dB) R SNR (dB) 
Ankle 
Xx 0.79 + 0.09 4.08 + 1.80 0.80 + 0.06 3.77 + 1.65 
y 0.86 + 0.11 6.25 + 2.66 0.82 + 0.18 5.31 43.42 
z 0.44 + 0.15 —0.20 + 1.48 0.30 + 0.11 -1.71+1.27 
Knee 
x 0.66 + 0.14 1.96 + 1.84 0.61 + 0.10 0.79 + 1.62 
y 0.79 + 0.13 4.28 + 2.02 0.70 + 0.14 2.50 + 2.49 
z 0.39 + 0.13 0.52 + 1.36 0.28 + 0.09 -1.80 + 1.09 
Angle 0.84 + 0.07 5.29 + 2.06 0.87 + 0.05 6.01 + 1.37 
Hip 
x 0.60 + 0.14 1.15 + 1.71 0.53 + 0.11 0.04 + 1.71 
y 0.66 + 0.14 1.97 + 1.92 0.42 +0.12 0.72 + 1.52 
z 0.56 + 0.13 0.66 + 1.75 0.63 + 0.09 1.20 + 1.67 
Angle 0.73 + 0.11 2.95 + 1.95 0.58 + 0.11 0.90 + 1.41 
Swing/stance 0.61 40.2 3.08 + 2.76 0.58 + 0.14 1.18 + 1.79 
Speed 0.42 + 0.26 -1.44 + 3.36 0.24 + 0.09 —2.03 + 0.85 
Step frequency 0.48 + 0.14 -1.84 42.77 0.57 + 0.22 0.93 + 1.84 
Stride length 0.40 + 0.28 -1.45 + 3.53 0.30 + 0.17 -1.89 + 1.64 


the distinction between M1 and S1 was not clear cut because S1 
modulations were capable of predicting motor parameters in the 
future (Figures 6E-H). 


GENERALIZATION PROPERTIES AND A SWITCHING MODEL 

A linear decoder trained on a single walking paradigm (e.g. 
forward walking only or backward walking only) was able to 
accurately extract walking parameters during the same paradigm 
(SNR = 5.29 dB for the forward model applied to forward walk- 
ing and SNR = 3.53 dB for the backward model applied to back- 
ward walking; Figures 7A, left and 7B, right). However, predictive 
power was not retained when applied to predict a different para- 
digm (SNR = —2.91 dB for the forward model applied to backward 
walking and SNR = —2.77 dB for the backward model applied to 
forward walking; Figures 7A,left panel and 7B,right panel). These 
results were confirmed by NDA (Figure 7D). 

At the same time, general models trained on a mixture of both 
forward and backward walking (walking forward 45.1 + 1.4% 
and backward 54.9 + 1.4% of the time) accurately predicted both 
forward (Figure 7C, left panel; SNR = 4.74 dB) and backward 
(Figure 7C, right panel; SNR = 2.24 dB) walking separately, albeit 
slightly worse compared to the single paradigm models. This was 
also evident from NDA (Figure 7D). 

To recapture the lost prediction accuracy when using general 
models versus single paradigm models, we generated a multi-layer 
switching model (Figures 7F-I). The switching model classified the 
subject’s walking as a number of sub-behaviors. That is, while the 
monkey was performing variable direction walking, walking was 
sub-classified into forward walking and backward walking periods. 
The switching model had two layers. The primary layer contained 


several independent submodels for kinematic variables, each one 
trained on data from a single sub-behavior. The secondary layer 
was a director layer that predicted the current behavioral mode 
(here, walking direction) for the subject and selected the appropri- 
ate kinematic submodel. The switching model outperformed the 
standard linear model trained during sessions with both forward 
and backward walking. The switching model improved the predic- 
tions of all kinematic variables on average by 13.8. 


PREDICTING MULTIPLE PARAMETERS SIMULTANEOUSLY 
Simultaneous extraction of many parameters essentially depended 
on the size of the neuronal population (Figures 8A—C). Random 
subpopulations were pooled from the entire neuronal ensemble and 
used to predict several parameters simultaneously. For each param- 
eter, prediction accuracy was normalized by its maximum value. 
The accuracy of simultaneous predictions was characterized by the 
normalized accuracy for the least well predicted parameter. Neuron 
dropping curves for this metric had less steep slopes when larger 
parameter sets were predicted (Figure 8A). In other words, smaller 
neuronal populations could predict a few parameters simultane- 
ously, and larger populations were required for predicting many 
parameters. This is illustrated in Figure 8B, which shows that the 
number of neurons required to achieve a given minimum accuracy 
(75%, 85% or 95% of maximum prediction accuracy) increased 
with the number of simultaneously predicted parameters. 
Similar to the result for the simultaneous extraction of mul- 
tiple parameters, larger neuronal populations were required to 
predict complex patterns of walking compared to more simple 
walking patterns. For example, the number of neurons required 
to achieve 95% of maximum prediction accuracy for the X 
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FIGURE 6 | Neuronal dropping curves for simultaneously extracted 
variables. Dots represent SNR for the extraction of walking parameters using 
models trained on randomly selected neuronal populations of different sizes. In 
(A-C) red dots correspond to X coordinate, green dots toY, and blue dots to Z. In 
(B,C) cyan dots correspond to joint angles for the respective joint. This analysis 
showed that prediction accuracy increased with the neuronal sample size [but 
see (D)]. Extraction analysis by cortical area was also performed for Monkey 1 
(E,G) and Monkey 2 (FH). In Monkey 1, more cells were recorded in M11 than in 
S1, whereas in Monkey 2 more cells were recorded in S1. (A) 3D coordinates of 
the ankle. (B) Joint angle and 3D coordinates of the knee. (C) Joint angle and 3D 
coordinates of the hip. (D) Foot contact, that is whether or not the monkey's foot 
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was on the ground. Foot contact was well predicted for smaller numbers of 
neurons but experienced overfitting for large numbers of neurons. (E,F) 


Neuronal dropping curves for the entire neuronal 
with extrapolated fits. (G,H) Prediction accuracy 
(blue curves) as a function of time lag between t 
time at which neuronal activity was sampled. In 


population and separate areas 
or M1 (red curves) and S1 

he time of prediction and the 
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neurons best predicted current kinematics using neuronal data in the past, 
indicating that neuronal modulations preceded motion. Monkey 2's postcentral 
neurons had a peak in predictive power using future data, indicating that activity 
was a replica of the kinematics. In Monkey 1, the postcentral subpopulation did 
not have particularly strong predictive power in this analysis. 


position of the ankle clearly increased when walking conditions 
of increasing complexity were employed (Figure 8C). On average, 
60 neurons were sufficient for predicting constant-speed walking 


in the forward direction. However, 90 neurons were needed to 
achieve this level of accuracy for variable-speed, forward walking. 
As complexity increased, more neurons were needed to maintain 
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FIGURE 7 | Generalized Training and the switching model. Panels A-C each 
represent a separate predictive model. In each, the left panel shows each 
model's ability to predict the kinematics of walking during sessions where the 
monkeys walk forward while the right panel shows each model's predictions of 
limb kinematics during periods of backward walking. The models accurately 
predicted conditions the same as the data they were trained upon but did not 
generalize well to conditions not contained within the training data set. The 
generalized model predicted either walking paradigm, but less accurately 
compared to paradigm-specific models. In response to these effects, a 
switching model (F-I) was designed that increased prediction accuracy for many 
kinematic parameters. (A) Actual and extracted traces of the X coordinate of the 
ankle for the forward walking model. Blue curves represent the actual position, 
and red curves represent the extracted position. (B) Traces for backward walking 
model. (C) Traces for generalized model. (D) Neuron dropping curves calculated 
by predicting forward walking using randomly selected neuronal populations 
with each model. (E) Neuron dropping curves calculated by predicting backward 
walking using randomly selected neuronal populations with each model. 
(F) Block diagram comparing the “switching” model with a generalized training 
model. Instead of using the same neuronal weights for all walking paradigms, 
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the switching model uses two separate kinematic submodels running in parallel. 
Each model is optimized for a separate walking paradigm: forward and backward 
walking. A state predictive model determines walking mode and selects which 
of the two kinematic submodels becomes the final output. (G) Comparison of 
prediction strength of individual neurons during forward (abscissa) versus 
backwards (ordinate) walking. The prediction strength (R) was determined for 
single-neuron predictions of the X position of the ankle. While there is a general 
correlation between the Rs for these walking paradigms, there is a clear 
subpopulation of neurons with high Rs for forward, but not for backward 
walking. (H) Representative records of the X position of the ankle illustrating the 
predictive power of the switching model (the panel highlighted in yellow) versus 
generalized model (red curve) during forward walking. Note that the specialized 
model selected by the state predictive model (green curve) matches the original 
record (black curve) better than the generalized model (red curve) and the 
submodel for backward walking (blue curve). (I) Representative records for 
backward walking. The backward paradigm submodel selected by the state 
predictor (blue curve in the panel highlighted in yellow) matches the actual 
position (black curves) better than the generalized model (red curve) and the 
submodel for forward walking (green curve). 
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complete, the binned neural activity is sent into the models and kinematic 
output is generated. This kinematic output is used to drive robotic actuators. 
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occurring and there was not yet any predicted output. In the bottom panel 
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the same prediction accuracy. Thus, 95 neurons were required for 
predicting backward walking at constant speed, while 115 neu- 
rons were required for predicting backward walking at variable 
speed. Predicting variable-speed bidirectional walking required 
110 neurons. 


REAL-TIME PREDICTIONS OF LEG KINEMATICS USING CORTICAL 
NEURONAL ENSEMBLES 

Real-time prediction of kinematic variables (Figure 8D) was per- 
formed in 22 daily recording sessions in Monkey 1. In each session, 
the first 5 min of walking were used for model training data. These 
data were sent to a buffer which was used to calculate a set of linear 
models for the leg kinematic parameters. These models were then 
used to generate real-time predictions from neural spike data alone. 
The actual and extracted joint angles are shown in Figure 8E, and 
the actual and extracted leg configurations are shown in Figure 8F. 
Real-time predictions were not statistically different from offline 
predictions (real-time SNR = 2.38 + 1.14, offline SNR 2.46 + 1.57, 
P> 0.05, Student’s t-test). The real-time system included a web link 
that could send the extracted kinematics to robotic appendages 
and return visual feedback (Figure 8D). The operation of robotic 
devices under these conditions has been reported in abstract form 
(Cheng et al., 2007; Lebedev et al., 2007) and will be covered in 
detail in a future article. 


DISCUSSION 

In this study, we extracted bipedal walking patterns from the 
modulations of discharge rates of monkey S1 and M1 neuronal 
ensembles. While such modulations were expected from pre- 
vious studies in quadrupeds (Armstrong and Marple-Horvat, 
1996; Beloozerova and Sirota, 1998; Drew, 1993; Prilutsky et al., 
2005; Widajewicz et al., 1994), it was not clear until the present 
study whether cortical activity would be sufficient for accurate 
predictions of leg kinematics and EMGs during bipedal walk- 
ing in primates. These findings have several implications. First, 
we have demonstrated the utility of using nonhuman primates 
for elucidating the neurophysiological mechanisms of bipedal 
locomotion. As such, we propose that chronic, multi-site cortical 
recordings could be used in the future to elucidate the neuronal 
mechanisms involved in upright posture control and bipedal 
walking. Second, as an implication for the BMI field, our find- 
ings provide the first proof of concept demonstration that, in the 
future, a neuroprosthetic for restoring bipedal walking in severely 
paralyzed patients can be implemented. 


CORTICAL NEURONAL MODULATIONS DURING BIPEDAL LOCOMOTION 

Using cortical activity rather than subcortical locomotion centers for 
extracting walking patterns is not without controversy (Prochazka 
et al., 2000). Although locomotion is conventionally recognized 
as a highly automated motor activity, subserved by spinal central 
pattern generators (Dietz, 2001; Grillner, 2006; Grillner et al., 2008; 
Hamm et al., 1999; Kiehn et al., 2008; McCrea and Rybak, 2008; 
Yamaguchi, 2004), the involvement of the cerebral cortex in gait 
control has been reported in studies conducted in rodents (Hermer- 
Vazquez et al., 2004), cats (Armstrong and Marple-Horvat, 1996; 
Beloozerova and Sirota, 1993), and monkeys (Courtine et al., 
2005b). Moreover, cortical involvement in locomotion seems to 


increase during more demanding tasks, especially those that require 
visual feedback (Armstrong and Marple-Horvat, 1996; Beloozerova 
and Sirota, 1993; Drew et al., 2008). 

Sampling from large populations of M1 and S1 neurons allowed 
us to analyze activity modulations in hundreds of simultaneously 
recorded neurons. Both M1 and S1 neurons modulated their firing 
rate in relationship to the stepping cycle. Firing for each neuron 
peaked at a particular phase of the stepping cycle. Given the large 
number of neurons that we recorded, it was unrealistic to examine 
in detail the properties of each neuron in order to determine which 
motor or sensory responses determined its step cycle modulations. 
Therefore, we used a population based approach in which cycle 
modulations of individual neurons each contributed to the extrac- 
tion of the step cycle at every time step, and more precise extractions 
were obtained by combining the information from many neurons. 
We also showed the benefits of utilizing multi-site recordings to 
predict locomotion kinematic parameters. This approach was 
beneficial because neuronal modulations from different cortical 
areas provided the diversity and richness of cortical signals needed 
for the accurate extraction of motor parameters under different 
behavioral conditions. In this context, we consistently observed 
superior performance of neuronal populations drawn from several 
cortical areas compared to the performance of populations drawn 
from a single area. 


EXTRACTING OF MULTIPLE PARAMETERS OF BIPEDAL WALKING 
Walking patterns were extracted using multiple linear models. As 
in previous studies in which predictions of arm movements were 
obtained, NDA showed that large neuronal ensembles were needed 
for accurate predictions of leg kinematic parameters. Thus, larger 
neuronal samples were required for simultaneous extraction of 
multiple walking parameters. Small populations of relatively spe- 
cialized neurons could accurately extract only a subset of locomo- 
tion parameters, and additional neurons were required to improve 
the extraction of the other parameters. As a corollary to this find- 
ing, larger neuronal populations were required as task complexity 
increased. These results support our previous suggestion that, to 
efficiently control a multiple degree of freedom neuroprosthetic, 
recording the activity of large ensembles of cortical neurons will 
be necessary (Carmena et al., 2003; Lebedev et al., 2007; Wessberg 
et al., 2000). 


M1 VERSUS S1 CONTRIBUTIONS TO THE PREDICTION OF LEG 
KINEMATICS 

Both M1 and S1 neurons contributed significantly to the predic- 
tions of leg kinematics. As expected, M1 modulations were more 
informative for predicting future values in the parameters of walk- 
ing, whereas S1 modulations better predicted the past values of 
the parameters. However, this distinction was not absolute, par- 
ticularly in Monkey 2 in which accurate predictions of future val- 
ues of locomotion parameters could be obtained from S1 activity. 
While repetitive nature of the movement may be one contributor 
to S1 predictive power, predictions obtained from S1 activity could 
also reflect centrally generated signals related to communication 
between motor and sensory areas and efferent copies (Chapin and 
Woodward, 1982; Fanselow and Nicolelis, 1999; Lebedev et al., 
1994). 
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GENERALIZED TRAINING 

The computational models of a BMI take advantage of correlations 
between neuronal firing modulations and behavioral parameters 
to generate their predictions. These correlations do not necessar- 
ily reflect a direct relationship or causation between neuronal dis- 
charges and the predicted parameters (Lebedev and Nicolelis, 2006). 
Consequently, a model trained under one set of conditions may not 
perform well when the conditions are changed and the correlation 
between the neuronal discharges and behavioral parameters are 
altered. We observed this effect when we attempted to predict dif- 
ferent locomotion patterns (Figure 7). Models trained on forward 
walking did not accurately predict backward walking and vice versa. 
This drop in prediction performance occurred because of the dif- 
ferences in neuronal modulations during the step cycles of forward 
walking versus backwards walking (Figure 4). Such an effect was 
expected because leg dynamics change dramatically during back- 
wards walking despite relatively small changes in leg kinematics 
(Thorstensson, 1986). It was only when the models were trained on 
a wider variety of patterns that they improved their ability to predict 
more diverse behaviors. Given that motor cortical neurons may 
represent muscle activity, limb kinematics and their combinations 
(Kakei et al., 1999), which all change in complex ways when behav- 
ioral paradigms are altered, this result is not entirely surprising. For 
instance, the pattern of muscle activation may become different for 
a given leg orientation in a new walking condition, and muscle-rep- 
resenting neurons would introduce errors in predicted kinematics. 
It is only when the linear models are trained on several behaviors 
that the kinematics can be accurately extracted. These results sug- 
gest that in order to build a neuroprosthetic that is both accurate 
and versatile in restoring locomotion patterns, it will be necessary 
to generate a training protocol which is varied enough so that the 
employed models learn to predict a large set of behaviors. 


SWITCHING MODEL 

While generalized training had the advantage of producing models 
that provided accurate predictions of the various behavioral para- 
digms included in the training data set, generalized models predicted 
with less accuracy than specific models designed for a single walking 
paradigm. To recapture some of this lost accuracy, we developed a 
switching model that selected the appropriate specific model once a 
switch in the walking paradigm was detected. Algorithms based on 
the idea of switching state spaces have been previously proposed for 
extracting arm movements from neuronal modulations (Kim et al., 
2003; Wu et al., 2004). While the switching model has many advan- 
tages and it increased the accuracy of predictions throughout this 
experiment, it does carry some potential drawbacks. For variables that 
do not behave differently between behavioral paradigms, there is little 
lost accuracy to recapture, and segregating the data in the training set 
into smaller sections for each submodel can have a detrimental effect, 
that is, the lower quantity of training data can produce a less accurate 
model. Additionally, the performance of the switching model is heav- 
ily dependent on the performance of the state predicting model that 
chooses which kinematic submodel is selected 


REAL-TIME BMI FOR REPRODUCTION OF LOCOMOTION PATTERNS 
Since one of the long-term goals of this research is to build a neu- 
roprosthetic device for the restoration of locomotion in paralyzed 


subjects, we tested the performance of our decoding algorithm 
using a real-time BMI suite that incorporated MAP recording 
hardware (Plexon Inc., TX, USA), a custom wireless video track- 
ing system and a set of real-time prediction algorithms. The BMI 
suite sampled the activity of up to 512 neuronal units and con- 
verted this neuronal activity into the predictions of multiple degree 
of freedom kinematic and behavioral parameters. The BMI suite 
streamed the predictions using an internet protocol. The successful 
implementation of this apparatus allowed us to obtain a proof of 
concept that a real-time BMI for the reproduction of locomotion 
can be implemented in the future as the core of a neuroprosthetic 
device aimed at restoring bipedal locomotion in severely paralyzed 
patients. 


RESTORATION OF LOCOMOTION BEHAVIORS USING A BMI 

Based on these results, we propose an approach to restore locomo- 
tion in patients with lower limb paralysis that relies on using cortical 
activity to generate locomotor patterns in an artificial actuator, 
such as a wearable exoskeleton (Figure 8G; Fleischer et al., 2006; 
Hesse et al., 2003; Veneman et al., 2007). This approach may be 
applicable to clinical cases in which the locomotion centers of the 
brain are intact, but cannot communicate with the spinal cord 
circuitry due to spinal cord injury. The feasibility of employing 
a cortically driven BMI for the restoration of gait is supported 
by fMRI studies in which cortical activation was detected when 
subjects imagined themselves walking (Bakker et al., 2007, 2008; 
Iseki et al., 2008; Jahn et al., 2004) and when paraplegic patients 
imagined foot and leg movements (Alkadhi et al., 2005; Cramer 
et al., 2005; Hotz-Boendermaker et al., 2008). Event-related poten- 
tials also demonstrated cortical activations in similar circumstances 
(Halder et al., 2006; Lacourse et al., 1999; Muller-Putz et al., 2007). 
Further support for this idea comes from recent studies of EEG- 
based brain-computer interfaces for navigation in a virtual environ- 
ment in healthy subjects (Pfurtscheller et al., 2006) and paraplegics 
(Enzinger et al., 2008). 

While a cortical BMI based neuroprosthesis that derived all its 
control signals from the user would have to cope with the lack 
of signals normally derived from subcortical centers, such as the 
cerebellum, basal ganglia and brainstem (Grillner, 2006; Grillner 
et al., 2008; Hultborn and Nielsen, 2007; Kagan and Shik, 2004; 
Matsuyama et al., 2004; Mori et al., 2000; Takakusaki, 2008), these 
problems may be avoided by an approach which only derives higher 
level leg movement signals from brain activity, while allowing 
robotic systems to produce a safer, optimum output. The challenge 
of efficient low-level control could be overcome by implementing 
“shared brain—machine” control (Kim et al., 2006), i.e. a control 
strategy that allows robotic controllers to efficiently supervise low- 
level details of motor execution, while brain derived signals are 
utilized to derive higher-order voluntary motor commands (step 
initiation, step length, leg orientation). 

A cortically driven BMI for the restoration of walking may 
become an integral part of other rehabilitation strategies 
employed to improve the quality of life of patients. In particular, 
it may supplement the strategy based on harnessing the remaining 
functionality and plasticity of spinal cord circuits isolated from 
the brain (Behrman et al., 2006; Dobkin et al., 1995; Grasso et al., 
2004; Harkema, 2001; Lunenburger et al., 2006). Indeed, cortically 
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driven exoskeletons may facilitate spinal cord plasticity, helping to 
recover locomotion automatisms. Additionally, cortically driven 
neuroprostheses may work in cohort with rehabilitation meth- 
ods based on functional electrical stimulation (FES; Hamid and 
Hayek, 2008; Nightingale et al., 2007; Wieler et al., 1999; Zhang 
and Zhu, 2007). In such an implementation, the BMI output 
could be connected to a FES system that stimulates the subject’s 
leg muscles. Finally, there is the intriguing possibility of connect- 
ing the BMI to an electrical stimulator implanted in the spinal 
cord, a strategy that may help induce plastic reorganization within 


these circuits. 


Altogether, our results indicate that direct linkages between the 
human brain and artificial devices may be utilized to define a series 
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Brain—machine interfaces (BMIs) establish direct communication between the brain and artificial 
actuators. As such, they hold considerable promise for restoring mobility and communication 
in patients suffering from severe body paralysis. To achieve this end, future BMIs must also 
provide a means for delivering sensory signals from the actuators back to the brain. Prosthetic 
sensation is needed so that neuroprostheses can be better perceived and controlled. Here 
we show that a direct intracortical input can be added to a BMI to instruct rhesus monkeys 
in choosing the direction of reaching movements generated by the BMI. Somatosensory 
instructions were provided to two monkeys operating the BMI using either: (a) vibrotactile 
stimulation of the monkey’s hands or (b) multi-channel intracortical microstimulation (ICMS) 
delivered to the primary somatosensory cortex (S1) in one monkey and posterior parietal 
cortex (PP) in the other. Stimulus delivery was contingent on the position of the computer 
cursor: the monkey placed it in the center of the screen to receive machine-brain recursive 
input. After 2 weeks of training, the same level of proficiency in utilizing somatosensory 
information was achieved with ICMS of S1 as with the stimulus delivered to the hand skin. 
ICMS of PP was not effective. These results indicate that direct, bi-directional communication 
between the brain and neuroprosthetic devices can be achieved through the combination of 
chronic multi-electrode recording and microstimulation of S1. We propose that in the future, 
bidirectional BMIs incorporating ICMS may become an effective paradigm for sensorizing 
neuroprosthetic devices. 


Keywords: brain-machine interface, primate, neuroprosthetics, cortical microstimulation, neuronal ensemble 


recordings 


INTRODUCTION 
During the last decade, considerable progress has been made in 
research on neuroprosthetics — devices that attempt to treat neuro- 
logical impairments and limb loss by interfacing intact neural struc- 
tures with artificial devices that enact motor or sensory functions 
(Nicolelis, 2003; Lebedev and Nicolelis, 2006; Schwartz et al., 2006; 
Fetz, 2007). Sensory neuroprosthetics, such as cochlear (Merzenich 
et al., 1974; Fallon et al., 2008) and visual (Dobelle et al., 1978; 
Dagnelie, 2008) implants, make use of electrical stimulation of 
sensory neural structures to recreate afferent input and perceptual 
experiences. Motor neuroprosthetics, often called BMIs (Nicolelis, 
2001), transform neuronal activity recorded in motor areas into 
commands to move artificial actuators such as cursors (Serruya 
et al., 2002; Taylor et al., 2002) and robotic devices (Wessberg et al., 
2000; Carmena et al., 2003; Velliste et al., 2008), and for the func- 
tional stimulation of muscles (Moritz et al., 2008). 

Curiously, sensory and motor neuroprosthetics have developed 
largely in parallel, without attempts to build an integrated system 
with both motor and sensory capabilities. Normal interactions with 


the environment, however, require conjoint processing of sensory 
and motor signals (Witney et al., 2004; Crapse and Sommer, 2008). 
Thus, the neurophysiological mechanisms underlying voluntary 
motor control in mammals are critically dependent on the affer- 
ent innervation of the skin, muscles and joints (Ribot-Ciscar et al., 
2003; James et al., 2007). Indeed, the paucity of sensory signals 
available in current BMIs could be a factor limiting their future 
clinical usefulness. 

Here we propose that future clinical neuroprostheses could be 
based on a bi-directional system in which motor control signals are 
extracted from the brain, using multi-electrode arrays implanted 
in motor areas, while tactile, proprioceptive and other useful sig- 
nals, are sent back to the brain through spatiotemporal patterns of 
intracortical microstimulation (ICMS) delivered to sensory areas 
(Lebedev and Nicolelis, 2006). To advance this goal, we added a 
direct intracortical input to a BMI, based on multi-electrode cortical 
microstimulation used by a rhesus monkey for enacting arm reach- 
ing movements (Figure 1A). This intracortical input instructed the 
direction of BMI-generated cursor movements. 
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FIGURE 1 | Schematic of the experimental paradigm. (A) A monkey is 
seated in front of a video screen to which the images of a cursor and visual 
targets are projected. The animal acquires targets with the cursor either 
manually using a joystick or directly by the activity of populations of 
motor-cortical cells. If more than one target is presented, reach direction is 
instructed by joystick vibration or artificial somatosensation delivered via ICMS 
of primary somatosensory cortex (S1). (B) Implanted sites in Monkey 1. The 
monkey was implanted with microwire arrays in two motorcortical areas 
(dorsal premotor, PMd, and primary motor, M1) and a primary somatosensory 
area (S1). (C) Zoomed-in region focuses on the S11 implant. Red circles 
indicate electrode pairs used for ICMS. Lines between electrodes indicate 
receptive field boundaries. The electrode grid covered the representation of 
digits D2-D5 and hand pads. (D) Illustration of the monkey palm with the 
shaded regions indicating the receptive fields for the electrode pairs used for 
ICMS. (E) Parameters of an ICMS pulse train. 


MATERIALS AND METHODS 

SUBJECTS 

Two adult male rhesus macaque monkeys (Macaca mulatta, 6.7 and 
6.5 kg) participated in this study. All animal procedures were per- 
formed in accordance with the National Research Council’s Guide for 


the Care and Use of Laboratory Animals and were approved by the 
Duke University Institutional Animal Care and Use Committee. 


CORTICAL IMPLANTS 
Each monkey received multiple microwire array implants in cortical 
areas, contralateral to their working hands. Aseptic surgery was per- 
formed to implant electrode arrays. Monkey 1 was implanted with 
six microelectrode arrays (32 microwires in each; Figures 1B,C) 
in primary motor cortex (M1), primary somatosensory cortex 
(S1) and dorsal premotor cortex (PMd) of both hemispheres. 
For the present study, we report only data from the arrays in the 
left hemisphere of Monkey 1. Monkey 2 was implanted with four 
32-microwire arrays in the right hemisphere: M1, PMd, posterior 
parietal cortex (PP) and supplementary motor area (SMA). 
Within each 32-channel array, electrodes were grouped into 
16 pairs. Each pair consisted of two microwires positioned tightly 
together. One electrode in each pair was 300 microns longer than its 
mate, with the longer electrode always having equal or larger diam- 
eter. The separation between adjacent pairs was 1 mm. Monkey 1 was 
implanted with polyimide coated tungsten electrodes, all 50 microns 
in diameter, in M1 and PMd and polyimide coated stainless steel 304 
electrodes with 40 and 63 micron diameters in S1. Monkey 2 was 
implanted with arrays of mixed tungsten and stainless steel 304 elec- 
trodes in M1 and SMA. The tungsten electrodes were 50 microns 
in diameter and coated with polyimide insulation; the stainless 
steel 304 electrodes were 45 microns in diameter and coated with 
polytetrafluoroethylene. Monkey 2 also received polyimide coated 
tungsten electrodes, all 50 microns in diameter, in PMd and PP. 
The PMd and M1 arrays were used to extract motor commands 
and the S1 array was employed as the target for ICMS (Figure 1C). 
In the second animal, motor signals were extracted from the SMA, 
PMd and M1 arrays, and ICMS was delivered to the PP array. The 
electrodes chosen for ICMS in Monkey 1 yielded recorded S1 neu- 
rons with clear receptive fields located on the ventral aspect of the 
second, third and fourth digits and palm pads (Figures 1C,D). No 
clear receptive fields were identified for the PP implant in Monkey 
2. In this animal, we used the same arrangement of the stimulation 
electrodes within the electrode grid as described for Monkey 1. 


NEURONAL RECORDINGS 

Extracellular spiking activity was amplified, digitized, and high- 
pass filtered using Multichannel Acquisition Processors (Plexon, 
Inc.). Action potentials were detected and sorted on-line with tem- 
plates set by the experimenter using Plexon spike-sorting software. 
The quality of the units was quantified by analyzing the refrac- 
tory period, estimated from the interspike intervals (ISIs). To be 
qualified as a single unit, in addition to having a distinct shape and 
amplitude (Nicolelis et al., 2003), at least 99.9% of its waveforms 
had to exhibit a refractory period greater than 1.6 ms (Hatsopoulos 
et al., 2004). Using these criteria 69.1% of the recorded units were 
single units, and 30.9% were classified as multi-unit neuronal activ- 
ity. Single and multi-units were treated equally during online BMI 
sessions and for offline analysis purposes. 


BASIC EXPERIMENTAL PROCEDURES 
Each monkey was seated in a primate chair and viewed a com- 
puter display. Their heads were unrestrained and the recording 
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system was connected to the implants using flexible wires. A two 
degree-of-freedom (left-right and forward-backwards) joystick was 
mounted at the monkey’s waist level. The monkeys were trained 
to manipulate the joystick with their hands. Monkey 1 performed 
with the right hand; Monkey 2 performed with the left hand. An 
electrical touch sensor on the joystick handle measured whether 
the monkey was holding the joystick. An LCD projector displayed 
visual images on a screen mounted 1.5 m in front of the monkeys. 
Using the joystick, the monkeys moved a circular cursor with a 
1.6-cm diameter. Forward, backward, rightward and leftward 
movements of the joystick translated to the upward, downward, 
rightward and leftward movements of the cursor, respectively. The 
joystick to cursor gain varied between 3.2 and 6.4 depending on 
the session (i.e., a 1-cm movement of the joystick translated into a 
3.2- to 6.4-cm movement of the cursor). Targets were defined by 
rings 4.8 to 5.6 cm in diameter. 


BEHAVIORAL TASKS 
Three behavioral tasks were employed (Figures 2A—C). The first 
two tasks, center-out (Figure 2A) and continuous target pursuit 
(Figure 2B), served to estimate basic parameters of the BMI such 
as prediction accuracy and neuronal tuning to motor parameters. 
The third task, target choice (Figure 2C), incorporated somato- 
sensory inputs. Each task required placing the computer cursor 
into a target using the joystick. Correct performance was rewarded 
by juice rewards. The center-out task used stationary targets that 
occurred at randomly chosen locations on a fixed-radius from the 
center of the screen. The monkey was required to hold the cursor 
at the central target and, once a peripheral target appeared, move 
the cursor to that target. 

The pursuit task used a moving target that followed a Lissajous 
curve as described by: 


x= Asin(avt+ 6) 


(1a) 


y= Bsin(bvt) (1b) 


where x and y are the x- and y-axis coordinates and tf is time in 
milliseconds. We used parameter values a = 3, b= 4, ve {0.00015, 
0.0002} kHz, 6 = 0.52, and A = B= 22.4 cm. 

The target choice task started with the monkey centering the 
cursor at a target that appeared in the middle of the screen. Then, an 
instruction period of 0.5—2 s followed. During this interval, a soma- 
tosensory stimulus was presented that instructed reach direction. In 
some experimental sessions, the stimulus instructed reaching to the 
right; in the others it instructed reaching to the left. If no stimulus 
was delivered, the monkey had to reach to the opposite direction. The 
stimulus itself was either a vibration of the joystick handle or ICMS. 
In some sessions, the stimulus was delivered for the entire instruc- 
tion interval; in others, it was delivered for a fraction of the total 
instruction interval. After the instruction period ended, the central 
target disappeared and two choice targets appeared on the screen. 
The monkey was rewarded for responding by placing the cursor over 
the target (right or left) corresponding to the direction indicated by 
the stimulus delivered during the instruction interval. 

In some sessions, we repeated the same stimulus in the trial 
immediately following one in which the monkey made an error. 
This was done for training purposes to prevent the monkeys from 


choosing the same target repeatedly without ever making reaches 
to the other target. However, when this rule was employed the 
monkeys could potentially make correct reaches in these trials 
without attending to the stimulus. Therefore, in sessions where 
this technique was employed, we restricted our analysis to trials 
that followed a rewarded trial and excluded those that followed 
non-rewarded trials. 


INTRACORTICAL MICROSTIMULATION 

Electrical stimulation was accomplished with a custom-built 
4-channel current-controlled, optically isolated stimulator. 
Simulation trains used in this study consisted of symmetric, bipha- 
sic, charge-balanced waveforms delivered synchronously and in 
a bipolar fashion through adjacent pairs of electrodes. Current 
amplitudes of 25-60 pA and pulse widths of 150-200 1s were used 
at frequencies between 30 and 60 Hz, depending on the session 
(Figure 1E). The initial current amplitudes used for ICMS of S1 
were determined by first finding the threshold current for evoked 
movements in M1 and then selecting a value just below that thresh- 
old. There was a 100-1s delay between the cathodic and anodic 
phases of the stimulation waveforms due to limitations with the 
stimulation hardware. 


LINEAR DISCRIMINANT ANALYSIS 

To classify reach direction during the instruction period of the 
target choice task, we used linear discriminant analysis (Fisher, 
1936). The interval from 1 s prior to the onset of the instruc- 
tion period to 1s after the offset was broken into 50 ms bins. 
The neuronal firing rate for each of the neurons in the ensemble 
was placed into a vector for each time-bin. This procedure was 
repeated for each trial. The trials were then partitioned into those 
in which the reach was made to the right and those where the reach 
was made to the left. From each of these sets, half of the trials 
were selected for training the classifier and half were selected for 
cross-validation. Using equal numbers of trials from each training 
set, linear classifiers were trained to discriminate reaches to the 
left and to the right for each time-bin. These classifiers, however, 
acted causally and cumulatively based on information accumu- 
lated from previous time-bins. This means that the classifier at 
time t= T used the output of the preceding classifiers (t< T) as 
well as the firing rates of the neurons at time t= T to classify 
reach direction. Performance of the classifier was measured with 
the cross-validation set. 


PREDICTION ALGORITHM 

Cursor position was reconstructed using multiple Wiener filter 
linear decoding algorithms applied to the population of recorded 
neurons (Wessberg et al., 2000; Haykin, 2002; Carmena et al., 2003; 
Lebedev et al., 2005). The filter represented the X and Ycoordinates 
of the joystick as a weighted sum of neuronal firing rates measured 
before the time of decoding: 


X()=b4+ > wyn,[t-—De] +e) (2) 


where X(t) is the value of the decoded parameter (for example, 
X-coordinate of the joystick) at time ¢, n, is the firing rate of neuron 
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FIGURE 2 |The three reaching tasks performed manually and through the 
BMI. (A) Center-out task. Monkeys first placed the cursor at a central location 
and then moved it to eccentrically located targets. (B) Continuous target pursuit 
task. Monkeys were rewarded for keeping the cursor within a continuously 
moving target. (C) Target choice task. Monkeys selected one of two targets. 
Directional instruction was provided by joystick vibration or intracortical 
microstimulation (ICMS) of S1. (D) Example traces for each of the tasks. Blue 
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traces are hand movements; red traces are predictions made from the neuronal 
ensemble activity. (E) Neuronal waveforms simultaneously recorded in three 
cortical areas (PMd, M1 and S1). (F) Directional tuning curves for the ensembles 
of cortical neurons recorded in Monkeys 1 (left) and 2 (right). Each horizontal line 
represents a neuron. Color represents firing rate normalized by standard 
deviation as a function of movement direction; neurons are sorted by preferred 
direction. 


i, Nis the total number of neurons, (j— 1)T is the time delay for tap 
h w,, 1s the weight for neuron i and tap j, b is the y-intercept and 
€(f) is the residual error. The word tap here simply refers to a time 
delay between the input (neuronal firing rate) and output (joystick 


movement) of the filter. The value of the tap length parameter T 
was set to 100 ms, and the number of taps was set to 10, that is 
neuronal rates were sampled in a 1000-ms window preceding the 
time of decoding. 
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To calculate model weights, w, Eq. 2 was converted to matrix 
form: 


X=NWte (3) 


where X is the matrix of joystick movements, N is the matrix of 
neuronal rates, W contains the weights of the model, € is the error. 
Each row of N corresponds to a specific time and each column is 
a vector of data for a particular neuron and time lag. Since our 
models took into account 10 lags, matrix N had 10 columns for 
every neuron. The y-intercept was calculated from a column of 
ones prepended to matrix N. We then solved for matrix W using 
the linear least squares approximation: 


W = inv(N’N)N'X (4) 


Each filter was first trained (i.e., the values of weights W were 
calculated) and then used as the decoder for new data. Accordingly, 
each experimental record was split into two halves: the training 
set and the validation set. The model was trained on the first set 
of the experimental data and predictions were obtained using the 
second set. Decoding was also conducted for the reverse arrange- 
ment: training the models on the second set and using it to predict 
the first set. 


DIRECT BRAIN CONTROL 

For each of the tasks, online, closed-loop experiments were per- 
formed. The monkeys first performed the behavioral tasks manually 
using a hand-held joystick (Hand Control; HC) and later control- 
led the cursor movements directly with their cortical activity. The 
behavioral and neural data, recorded during 10-30 min training 
epochs during which the monkeys moved the joystick manually, 
were used to train multiple linear decoding models that extracted 
cursor position from the modulations of cortical cells, as described 
above. Once the model weights were calculated, the mode of opera- 
tion was switched to brain control during which the joystick was 
disconnected from the cursor, and cursor position was directly 
controlled by the signals extracted from the brain. For some sessions 
the joystick was immediately removed upon transition to brain 
control (Brain Control Without Hand movements; BCWOH); for 
other sessions, the joystick was disconnected from the task control 
but otherwise left alone and the monkey was permitted to continue 
to move its hand (Brain Control With Hand movements; BCWH). 
For still other sessions, brain control was performed in both BCWH 
and BCWOH modes. 


RESULTS 

In this study, a somatosensory component was added to our previ- 
ous design of a BMI enacting arm reaching movements (Carmena 
et al., 2003). This component instructed the direction of reaching 
movements. Motor output of the BMI was controlled by the activ- 
ity of neuronal ensembles recorded in the primary motor (M1) 
and dorsal premotor (PMd) cortex (Figure 1B). The behavioral 
tasks consisted of acquiring visual targets with a computer cursor 
(Figures 1A and 2). Monkeys first performed the behavioral tasks 
manually using a hand-held joystick and then switched to con- 
trolling the cursor directly by their cortical activity. Predictions of 
cursor movement were generated using linear models (Wessberg 
et al., 2000; Carmena et al., 2003). Two types of somatosensory 


input were employed: (a) vibrotactile stimulation of the monkeys’ 
palms via a vibrator embedded in the joystick handle and (b) ICMS 
of primary somatosensory (S1) cortex in one monkey (Figure 1E) 
and the arm reach area of posterior parietal cortex (PP) in the 
other monkey. Both monkeys learned to use vibrotactile stimula- 
tion cues to instruct the direction of their cursor movements. The 
first monkey learned manual and BMI tasks instructed by ICMS 
of S1; the other was unable to learn to use ICMS of PP. 


ENSEMBLE ACTIVITY DURING REACHING TASKS 

We recorded simultaneously from ensembles of 50-200 corti- 
cal neurons, depending on the experimental session and subject 
(Figure 2E). Analysis of directional tuning indicated that the pre- 
ferred directions of the recorded neurons covered the entire range 
of angles from 0 to 360° (Figure 2F). Concurrent with record- 
ings during behavioral tasks, we performed periodic mapping 
of receptive fields of the neurons recorded in $1. These neurons 
revealed receptive fields distributed primarily on the hand and 
digits (Figures 1C,D). 


DECODING MOVEMENT 

The accuracy of predictions of hand position from cortical ensem- 
ble activity observed was in the same range as has been achieved 
in previous studies carried out in our laboratory (Wessberg et al., 
2000) and by other groups (Taylor et al., 2002). The prediction 
accuracy was higher in the early recording sessions, when the major- 
ity of implanted electrodes yielded the most high quality units and 
later decreased as some of the channels lost unit activity. Table 1 
shows the average performance of the predictions under different 
conditions. Across all tasks and sessions, when performing in HC 
mode, the average correlation coefficient (R), between the actual 
and predicted values of the joystick X-position, was 0.47 + 0.19 
(mean + standard deviation) for Monkey 1 and 0.53 +0.14 for 
Monkey 2. The R for Y-position was 0.42 + 0.18 and 0.51 + 0.17 
for Monkeys 1 and 2, respectively. 


Table 1 | Average prediction accuracy. Correlation coefficient between the 
actual hand movements and movements as predicted from cortical activity 
across different behavioral conditions for two monkeys. Sessions with brain 
control with hand movements (BCWH) are shown split into the initial hand 
control portion, when the online filter was trained, and the subsequent brain 
control portion. The mean + standard deviation is presented. 


Session type Monkey 1 Monkey 2 
HC N (sessions) = 139 N=78 
x 0.47 + 0.19 0.53 + 0.14 
y 0.42 +0.18 0.51 +0.17 
BCWH N=20 N=21 
Hand control portion 
x 0.57 + 0.20 0.58 + 0.14 
y 0.60 + 0.20 0.45 + 0.19 
Brain control portion 
Xx 0.41 +0.19 0.48 + 0.16 
y 0.50 + 0.27 0.38 +0.21 


Frontiers in Integrative Neuroscience 


www. frontiersin.org 


September 2009 | Volume 3 | Article 20 | 5 


349 


O'Doherty et al. 


Microstimulation instructs brain-machine interface 


The correspondence between the actual and predicted hand 
position decreased in sessions where the monkeys were permit- 
ted to continue to move their hands during direct brain control 
(BCWH; Table 1). The R for X-position decreased 28.1% (P< 0.01 
Wilcoxon signed-rank test) and 17.2% in Monkey 2 (P< 0.001). 
The R for Y-position decreased 16.7% (P< 0.01) and 15.6% 
(P< 0.005) in Monkeys 1 and 2, respectively. This decrease indi- 
cates that the neuronal ensemble adapted to controlling the cur- 
sor movements and became less representative of the animal’s 
hand movements — a finding that we and others have previously 
interpreted as evidence of the external actuator becoming incor- 
porated into the brain’s internal representation (Lebedev et al., 
2005; Tkach et al., 2007). 


SOMATOSENSORY STIMULUS 

Incorporation of somatosensory input into the BMI was investi- 
gated in the target choice task (Figure 2C). The monkeys chose 
between two visually identical targets based on a somatosensory 
input (vibration or ICMS), the presence or absence of which 
instructed reach direction (right or left identical circles; Figure 2C). 
In initial experiments with Monkey 1, the presence of a stimu- 
lus instructed reaches to the right, and the absence of stimulus 
instructed reaches to the left. This contingency was later reversed. 
Monkey 2 reached to the left if the stimulus was presented and 
reached to the right if it was absent. 

Monkey 1 was initially trained with vibrotactile stimulation and 
was able to perform at 90% accuracy after 12 sessions (Figure 3A). 
After 15 sessions of training in this task, the vibration cue was 
replaced with ICMS delivered to $1. However, the previous training 
with vibrotactile stimulation did not immediately transfer to the 
ICMS task. Rather, the monkey started with chance performance 
and began to improve only on the 15th session. After this point, 
learning was very rapid and the monkey achieved 90% accuracy 
by the 20th session. (Figure 3B). 

However, even before rapid behavioral improvement occurred 
in the ICMS task, the monkey’s performance was marginally, but 
statistically, above chance (60.02% versus 52.5% in control trials 
without microstimulation; P < 0.001, Wilcoxon ranked-sum test). 
This effect could reflect a bias in the monkey’s reach decision that 
resulted from the activation of motor preparatory circuits in the 
stimulated hemisphere (Hanks et al., 2006). If so, this bias was not 
crucial for task performance because the monkey learned a reverse 
contingency in which the instructed target was ipsilateral to the 
stimulated hemisphere (Figure 3C). 

Psychometric measurements in which ICMS current ampli- 
tude was varied on different trials showed that Monkey 1 could 
respond to currents as low as 25 1A (with a pulse width of 150 1s) 
and reached 90% accuracy when ICMS amplitude was at 50 pA 
(Figure 3D). The average proportion of correctly performed trials 
with an ICMS amplitude greater than 20 pA was 88%, which was 
significantly different from chance (P< 0.001, Wilcoxon ranked- 
sum test). For currents less than 20 pA the monkey performed 
at 50% chance level (P = 0.8). Curiously, Monkey 1 exhibited 
better performance when ICMS amplitude exceeded 20 pA in 
the trials immediately following error trials in which the instruc- 
tion simply repeated. In these trials, the monkey could receive 
a reward by simply changing the previous erroneous direction, 


so the instruction, in principle, could be ignored. Nonetheless, 
for ICMS currents less than 20 pA Monkey 1 performed cor- 
rectly in only 72% of trials, which was significantly different 
from chance (P< 0.001), but significantly worse (P< 0.001) than 
the 84% correct performance with ICMS currents greater than 
20 pA. Thus, the ICMS cue clearly benefited performance even in 
those trials. This effect was also clear from an analysis of reaction 
times. The reaction time on regular trials (i.e., rewarded trials 
preceded by rewarded trials) was 457 + 43 ms (mean + standard 
error) for the currents less than 20 pA and 454 + 18 ms for the 
higher currents. The small 3 ms average difference was statisti- 
cally significant (P< 0.001, Wilcoxon ranked-sum test). In the 
trials following error trials, the monkey’s decisions occurred sig- 
nificantly slower than on regular trials (P< 0.001), as indicated 
by increased reaction times: 785 + 60 ms for currents less than 
20 pA and 562 + 51 for those higher. Thus, higher ICMS currents 
resulted in a considerable (223-ms; P< 0.001) improvement in 
the reaction time. 

Monkey 2 learned the vibrotactile cued task on a similar time- 
course as Monkey 1 (7 sessions to start learning and 11 sessions 
to reach >90% accuracy; Figure 3A). This animal did not have 
an implant in $1 and was unable to learn the task instructed by 
ICMS of the PP even though training with ICMS continued for 
30 sessions. 


BMI WITH SOMATOSENSORY INPUT 

In the BMI with somatosensory input, one monkey controlled 
cursor movements directly by using motor cortical activity while 
receiving somatosensory instructive signals in the form of S1 
ICMS. The second monkey also controlled the cursor using motor 
cortical activity but, since PP ICMS was ineffective, received soma- 
tosensory signals via vibrotactile stimulation of the hand. The 
major technical challenge for carrying out these experiments was 
obtaining simultaneous recordings from large neuronal popula- 
tions in motor cortical areas concurrent with ICMS to sensory 
cortical areas. The potential for ICMS pulses occluding spikes 
was limited to 6-12% of the stimulation interval because electri- 
cal artifacts evoked by the pulses were brief (<2 ms) and easily 
distinguishable from action potentials. While we recorded neu- 
ronal activity during ICMS, we excluded the instruction intervals 
from the dataset used to train the BMI algorithms. This was done 
to avoid any influence of the stimulation on the parameters of 
the extraction algorithm (e.g., occluded spikes interpreted as a 
decrease in firing rate or neuronal responses to ICMS confused 
with motor commands). No neuronal activity was excluded dur- 
ing online BMI operations. 

Neuronal modulations were observed both during the 
instruction and movement periods of the target choice task. 
We observed neurons that exhibited increases and decreases in 
their firing rates during the instruction period, as well as neurons 
that were directionally selective exclusively during the movement 
period (Figure 3E). Movement direction could be predicted 
from neuronal population activity before the monkey moved 
the joystick (Figure 3F). In the early trials, when the monkey 
performed close to chance level (blue trace), the rise in the pre- 
dictions occurred before the delay. This suggests that the monkey 
started to form its decision before the stimulus was delivered. 
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FIGURE 3 | Learning to produce vibratory and ICMS cued reaches. 

(A) Improvement in target discrimination performance where somatosensory 
information was delivered via vibrotactile stimulation. Circles indicate Monkey 1; 
squares indicate Monkey 2. Filled symbols indicate statistically significant 
difference from chance performance (P< 0.01), open symbols correspond to 
chance performance. Solid lines indicate sigmoidal fits. (B) Improvement in 
target discrimination where somatosensory information was delivered via ICMS 
of Monkey 1. Solid line indicates sigmoidal fit. (C) Improvement in target 
discrimination in Monkey 1 after a reversal of the association between the ICMS 
and the response target. Solid line indicates linear fit. (D) Psychometric curve of 
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behavioral performance as ICMS pulse train amplitude was modulated. Solid line 
indicates sigmoidal fit. (E) Peri-event time histograms and rasters of the activity 
of four single units from M1 of Monkey 1. Trials are aligned on cue onset. Vertical 
red lines indicate start and end of the cue interval. Left columns indicate ICMS 
trials during which the monkey moved to the left; right columns indicate 
movements to the right. (F) Average linear discriminant analysis predictions of 
movement direction as a function of time since cue onset show learning-related 
changes. Blue and red traces indicate averages across early (n = 15) and late 

(n = 15) sessions, respectively; control sessions (n = 2) are shown in black. 
Vertical red lines indicate start and end of the cue interval. 


In other words, the animal was simply guessing. However, after 
learning (red trace, Figure 3F), the rise in neuronal prediction 
occurred subsequent to the instruction period onset and had a 
steeper slope, indicating a more rapid accumulation of evidence 


for the reach direction. In control sessions in which no ICMS 
was presented (black trace, Figure 3F), information about the 
impending movement was not present in the population activity 
until 0.5 s before movement onset. This indicates that in the late 
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sessions, the monkey did not make premature guesses. Instead, 
the evidence suggests that this monkey formed its target choices 
based on the pattern ICMS. 

Real-time, direct brain control was instructed by ICMS in 
sessions with Monkey 1 (Figure 4B) and by vibrotactile stimu- 
lation in sessions with Monkey 2 (Figure 4A). Monkey 1 per- 
formed without using the joystick in some sessions (Figure 4C); 


Monkey 2 continued to use the joystick (which was needed to 
produce vibrotactile stimulation). Both monkeys exceeded 70% 
correct performance in brain control. Monkey 1 performed above 
this level in seven sessions of brain control with ICMS. Monkey 2 
started by performing at a 55% correct performance level in brain 
control with vibrotactile input, and reached a greater than 80% 
performance on the seventh session of training. 
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FIGURE 4 | BMI operations. (A) The component of cursor position during the 
target choice task, illustrating manual control (left) and BMI control (right) from a 
single session with Monkey 2. Gray rectangles represent the instruction period; 
green rectangles represent the location of the correct target for each trial, its 
onset and offset time (horizontal dimension) and size (vertical dimension). 

(B) Same as panel A, but depicting a session with Monkey 1. (C) Average 
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movements made by Monkey 1 in the left and right directions for hand control, 
HC (left), BMI control with hand movements, BCWH (middle) and BMI control 
without hand movements, BCWOH (right). Trajectories are aligned on the time 
of target acquisition. Solid trace indicates mean cursor movements and dashed 
trace indicates mean hand movements. Shaded region is standard deviation of 
cursor movements. 
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DISCUSSION 

Altogether, these experiments demonstrate that somatosensory 
input can be incorporated in BMIs using either vibrotactile stimu- 
lation or ICMS of S1. This type of recursive somatosensory input 
can be considered a form of feedback because the stimulus was 
contingent on cursor position. The notion that electrical stimula- 
tion of cortical areas may reproduce physiological sensations dates 
back to the pioneering studies of Penfield (Penfield and Boldrey, 
1937). More recently, ICMS of S1 proved to be efficient to reproduce 
temporal (Romo et al., 1998) and spatiotemporal (Fitzsimmons 
et al., 2007) parameters of somatosensory inputs. Our findings are 
consistent with our previous demonstration that spatiotemporal 
patterns of ICMS delivered through chronically implanted multi- 
electrode arrays are viable over long intervals, leading to steady 
improvement of the behaviors instructed by ICMS (Fitzsimmons 
et al., 2007). 

As in our previous study (Fitzsimmons et al., 2007), we did 
not observe an immediate ability to interpret ICMS applied to S1 
in the monkeys previously overtrained with vibrotactile stimuli, 
which indicates a substantial difference between the sensations 
evoked by skin vibration and ICMS. This result is different from 
the description of Romo et al. (1998) who reported almost imme- 
diate learning with ICMS of S1. One explanation for this discrep- 
ancy is the specific S1 subdivision implanted. Romo et al. (1998) 
implanted area 3b, which is upstream to area 1 implanted in our 
study. Additionally, their experiment included a number of initial 
test sessions during which learning to discriminate ICMS could 
have occurred. 

When Monkey 1 finally did start learning the ICMS cue, its 
learning took a very rapid time course (Figure 3B). One possible 
interpretation is that the sensations evoked by ICMS of area 1 
were quite unlike skin vibration. Another explanation is that no 
sensations were initially evoked and that repeated exposure to 
ICMS was required for any sensation to develop. Further experi- 
ments will be required to better understand factors involved in 
the development of sensation due to S1 ICMS. Although ICMS 
of area 1 may not initially evoke sensations very similar to skin 
vibration, as follows from our present study and previous work 
(Fitzsimmons et al., 2007), this does not at all preclude its use in 
BMI applications. 

The action of $1 ICMS was not as trivial as stimulating M1 
through the direct projections from S1 to M1. M1 recordings 
did not reveal any ICMS-evoked responses per se in this corti- 
cal area. The presence of such evoked responses would be con- 
sistent with the animal’s behavioral response being mediated by 
a simple detection-like signal, caused by ICMS, migrating from 
S1 to M1. Instead, we observed the occurrence of a gradual build 
up of preparatory neuronal activity that did not merely mirror 
the constant frequency of ICMS (Figures 3E,F). Moreover, there 
was a 33% reaction time difference observed between low ICMS 
current amplitudes (<20 pA) and high ICMS current amplitudes 
(>20 pA) for correction trials in which the monkey knew the cor- 
rect response. This difference provides further evidence that the 
monkey was attending to the ICMS cue. Finally, the monkey suc- 
cessfully learned an ICMS-reach association reversal from a target 
contralateral to the stimulated hemisphere to a target ipsilateral to 
the stimulus (Figure 3C). This would not happen if the monkey 


simply reached toward the hemifield contralateral to M1 which 
was excited by the projections from S1. 

The second monkey was unable to learn to discriminate ICMS 
of PP, even after 30 daily sessions. This result mirrored our previ- 
ous observation in an owl monkey that was unable to learn to 
discriminate PP stimulation but later learned to discriminate ICMS 
delivered to $1 (Fitzsimmons et al., 2007). Therefore, it is conceiv- 
able that PP cannot be used for this type of sensory instruction or 
that use of PP may require much longer training or require dif- 
ferent parameters. This result suggests that for ICMS to instruct 
target selection, it may not be sufficient to simply inject current 
to any part of the cortical network that controls reaching move- 
ments, for example PP. Stimulation of primary sensory areas of 
the cortex (and possibly thalamus) appears to be most effective 
for sensory substitution, as follows from our present study, as well 
as previous studies by us (Fitzsimmons et al., 2007) and others 
(Richer et al., 1993). 

Several recent studies reported adding vibrotactile sensory chan- 
nels to EEG-based BMIs (Chatterjee et al., 2007; Cincotti et al., 
2007). While sensory substitution using haptic devices (Vidal- 
Verdu and Hafez, 2007) is clearly an important development that 
will benefit patients, sensorizing BMIs with ICMS is potentially 
capable of surpassing sensory substitution. Indeed, by using ICMS 
to directly target the cortical areas deprived of sensory input after 
neurological injury, disease or limb loss, we endeavor to directly use 
intact brain functionality rather than delegating sensory processing 
to other structures. 

The present demonstration of a BMI instructed by ICMS has 
brought together two experimental paradigms that were previ- 
ously studied in isolation: real-time BMI control based on neu- 
ronal ensemble recordings from cortical motor areas (Wessberg 
et al., 2000; Serruya et al., 2002; Taylor et al., 2002; Carmenaet al., 
2003; Moritz et al., 2008; Velliste et al., 2008) and ICMS of S1 as 
a means to evoke somatic sensations (Romo et al., 1998; Talwar 
et al., 2002; Butovas and Schwarz, 2007; Fitzsimmons et al., 2007; 
Huber et al., 2008; London et al., 2008). Our results show that, 
with training, monkeys can learn to select their reaching move- 
ments using somatosensory cues, and one monkey reported here 
learned to utilize S1 ICMS to mediate BMI-controlled reaching 
movements. We have established the feasibility of moving from 
a brain-machine interface to a brain-machine-brain interface in 
which artificial actuators and brain tissue are directly connected 
in a bidirectional way. These results lead us to suggest that in 
the future, ICMS delivered through chronically implanted multi- 
electrode arrays may be incorporated into many BMI paradigms 
and used for delivering artificial sensory feedback directly to the 
subject’s brain. 
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Brain-machine interfaces” use neuronal activity recorded from the 
brain to establish direct communication with external actuators, 
such as prosthetic arms. It is hoped that brain-machine interfaces 
can be used to restore the normal sensorimotor functions of the 
limbs, but so far they have lacked tactile sensation. Here we report 
the operation of a brain-machine-brain interface (BMBI) that both 
controls the exploratory reaching movements of an actuator and 
allows signalling of artificial tactile feedback through intracortical 
microstimulation (ICMS) of the primary somatosensory cortex. 
Monkeys performed an active exploration task in which an actuator 
(a computer cursor or a virtual-reality arm) was moved using a 
BMBI that derived motor commands from neuronal ensemble 
activity recorded in the primary motor cortex. ICMS feedback 
occurred whenever the actuator touched virtual objects. Temporal 
patterns of ICMS encoded the artificial tactile properties of each 
object. Neuronal recordings and ICMS epochs were temporally 
multiplexed to avoid interference. Two monkeys operated this 
BMBI to search for and distinguish one of three visually identical 
objects, using the virtual-reality arm to identify the unique artificial 
texture associated with each. These results suggest that clinical 
motor neuroprostheses might benefit from the addition of ICMS 
feedback to generate artificial somatic perceptions associated with 
mechanical, robotic or even virtual prostheses. 

Brain-—machine interfaces (BMIs) have evolved from 1-d.f. systems* 
to many-d.f. robotic arms* and muscle stimulators’ that perform com- 
plex limb movements, such as reaching®* and grasping’. However, 
somatosensory feedback, which is essential for dexterous control'*”’, 
remains underdeveloped in BMIs. With the exception of a few studies 
combining BMIs with tactile stimuli applied to the body’, existing 
systems rely almost exclusively on visual feedback. Prosthetic sen- 
sation has been studied in the context of sensory substitution'’* and 
targeted reinnervation’; however, these approaches have limited 
application range and channel capacity. To provide a proof-of-concept 
method of equipping neuroprostheses with sensory capabilities, we 
implemented a BMBI that extracts movement commands from the 
motor areas of the brain while delivering ICMS feedback in somato- 
sensory areas'’*'® to evoke discriminable percepts'’”°. This idea 
received support from our pilot study’®, in which a monkey responded 
to ICMS cues with the movements of a BMI-controlled cursor. 
However, the ICMS cue did not provide feedback of object-actuator 
interactions in this previous demonstration. 

The BMBI developed here allowed active tactile exploration” during 
BMI control (Fig. la). Two monkeys (M and N) received multielec- 
trode implants in the primary motor cortex (M1) and the primary 
somatosensory cortex (S1) (Fig. 1b). They explored virtual objects 
using either a computer cursor or a virtual image (avatar) of an arm 
(Supplementary Fig. 1a, b). In ‘hand control’, the monkeys moved a 
joystick with their left hands to position the actuator. They searched 
through a set of virtual objects, selected one with a particular artificial 


texture conveyed by ICMS, and held the actuator over that object to 
obtain reward (Fig. la and Supplementary Fig. 1c, d). During ‘brain 
control’, the joystick was disconnected and the actuator was controlled 
by the activity of right-hemisphere M1 neurons””*”*. The behavioural 
tasks varied in the number of objects on the screen, the artificial tex- 
tures used and the actuator type (Fig. 2a), and were more difficult than 
previously reported BMI tasks because of the presence of multiple 
objects in the workspace, a prolonged object selection period and the 
necessity of interpreting ICMS feedback. 

ICMS was delivered through two pairs of microwires to the hand 
representation area of S1 in monkey M (Fig. 1c) and through one pair 
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Figure 1 | The brain-machine-brain interface. a, Movement intentions are 
decoded from M1; artificial tactile feedback is delivered to S1. b, Microwires 
were implanted in M1 and S1. c, Microwires used for ICMS in monkey M are 
accented in red. d, Actuator movements for a trial in which monkey M explores 
UAT but ultimately selects RAT. Grey bars indicate stimulation patterns; insets 
indicate the ICMS frequency. e, Rastergram of M1 neurons recorded during the 
trial shown in d. 
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of microwires to the leg representation in monkey N. Each artificial 
texture consisted of a high-frequency pulse train presented in packets 
at a lower, secondary, frequency (Fig. 1d and Supplementary Fig. 2a). 
The rewarded artificial texture (RAT) consisted of 200-Hz pulse trains 
delivered in 10-Hz packets. The comparison artificial textures com- 
prised 400-Hz pulse trains delivered in 5-Hz packets (unrewarded 
artificial texture (UAT)) or an absence of ICMS (null artificial texture 
(NAT)). 

The main challenge solved here was the real-time coupling of ICMS 
feedback to the BMI decoder. Because ICMS artefacts masked neur- 
onal activity for 5-10 ms after each pulse (Fig. 1d, e), we multiplexed 
neuronal recordings and ICMS with a 20-Hz clock rate (Supplemen- 
tary Fig. 2a). The interleaved intervals proved adequate for online 
motor control and artificial sensation—a result that was not clear a 
priori because S1 stimulation could have affected M1 processing 
through the connections between these areas. 

BMBI performance improved with training. In task I (Fig. 2a), 
monkey M surpassed chance performance after nine sessions and 
monkey N did so after four sessions (P< 0.001, one-sided binomial 
test). Improvement continued with more difficult tasks (tasks II-V) 
(Fig. 2a, b and Supplementary Fig. 3a). In particular, the time spent 
exploring unrewarded artificial textures decreased (Fig. 2c and Sup- 
plementary Fig. 3b). Additionally, performance improved over the 
course of daily experimental sessions (Fig. 2d). Psychometric analysis 
of RAT stimulation amplitudes (Supplementary Fig. 2b) indicated that 
at least 8 nC per ICMS waveform phase (100-j1s-wide current pulses of 
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Figure 2 | Learning to use ICMS feedback. a, Behavioural tasks. 2D, two- 
dimensional; VR, virtual reality. b, c, Performance of monkey M (72 sessions). 
Circles (b) depict the fraction of correctly performed trials. Open circles indicate 
chance performance. Curves are lines of best fit. The asterisk indicates sessions 
used for psychometric measurements. Squares, triangles and crosses 

(c) represent mean times spent in RAT, UAT and NAT, respectively. Black, hand 
control; red, brain control. d, Intrasession performance for monkey M. Curves 
represent averages for brain control without hand movements (BCWOH; main 
panel, three sessions), for hand control (inset, 12 sessions) and for brain control 
with hand movements (BCWH;; inset, 12 sessions). Lines are best linear fits. 
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80 uA) was needed for the discrimination of artificial textures 
(P < 0.001, one-sided binomial test). Performance was at chance level 
for catch trials (task IT), where ICMS was not delivered (P = 0.90, one- 
sided binomial test). 

The statistics of object exploration intervals (total time spent over a 
particular object in a given trial) indicated that the monkeys uniquely 
discriminated each type of artificial texture (Figs 2c and 3a, c) and 
interpreted ICMS within hundreds of milliseconds—a timescale com- 
parable to that for the discrimination of peripheral tactile stimuli**”’. 
Early in task I, exploration intervals were equal for RAT and NAT 
(P > 0.5, Wilcoxon signed-rank test); with training, they became longer 
for RAT and shorter for NAT (tasks I and II) and UAT (tasks III-V). 
During hand control, the mean interval was longest for RAT (monkey 
M: 1,396 + 21 ms; monkey N: 1,165 + 15 ms; mean + s.e.m.), shortest 
for NAT (304+8ms; 300+10ms) and intermediate for UAT 
(452 + 13 ms; 402 + 14ms) (P<0.01, analysis of variance). During 
brain control, intervals spent exploring NAT (498+ 15 ms; 
587 + 25 ms) and UAT (685 + 20 ms; 764 + 32 ms) were longer than 
they were during hand control, but were still shorter than those spent 
exploring RAT (1,420 + 28 ms; 1,398 + 55 ms) (P< 0.01, analysis of 
variance). 

Additional hallmarks of active exploration were seen in the con- 
ditional probabilities of selecting different artificial textures (Fig. 3b, 
d). During hand control trials, the monkeys stayed over the first- 
encountered artificial texture (arrows that loop back to the same 
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Figure 3 | Statistics of object exploration. a, Object exploration intervals 
during hand control and brain control (inset) for monkey M (hand control: 
n = 1,809 trials; brain control: n = 1,355 trials). b, State transition diagrams for 
monkey M, indicating the probabilities of reaching among RAT, UAT and 
NAT after the first (left subpanel) or second (right subpanel) reach. Black labels, 
hand control; red labels, brain control; line thickness is proportional to 
transition probability. c, Same as a, but for monkey N (hand control: n = 808 
trials; brain control: n = 729 trials). d, Same as b, but for monkey N. 
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artificial texture in Fig. 3b, d) with high probability if it was RAT 
(monkey M: P = 0.70; monkey N: P = 0.76), but with low probability 
ifit was UAT (P = 0.05; P = 0.01) or NAT (P = 0.0; P = 0.0) (Fig. 3b, 
d, left). After examining the second artificial texture, the monkeys 
could identify the correct artificial texture either by apprehending it 
directly or through a process of elimination. This follows from the 
increase from chance to approximately P = 0.7 in the probability of 
moving to RAT from NAT or UAT and the decrease to P ~ 0.2 in the 
probability of revisiting UAT or NAT (Fig. 3b, d, right). Similar effects 
were observed for brain control (Fig. 3b, d, red text). 

Brain control started in task IV. During BCWH, the monkeys 
continued to hold the joystick although it was disconnected’*”. 
During BCWOH”™”, the joystick was removed. In monkey M, with 
more than 200 recorded neurons, performance was less accurate dur- 
ing BCWH (73.75 + 3.00%; mean + s.e.m.) than during hand control 
(91.48 + 1.20%). In monkey N, with 50 recorded neurons, perform- 
ance dropped further (50.37 + 3.74% versus 91.45 + 1.91%), but still 
significantly exceeded the 33% chance level. M1 neurons showed 
directionally tuned modulations (Supplementary Figs 5 and 6) that 
were retained across different interfering ICMS patterns during both 
hand control (Supplementary Fig. 4a, b) and brain control (Fig. 4a, b). 
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Figure 4 | M1 modulations during active control versus passive observation. 
a, b, Average brain control movements for monkey M (n = 294 trials) towards 
the RAT, UAT and NAT objects appearing on the left-hand side of the screen 
(a), and corresponding neuronal modulations (b). The colour scale shows 
normalized firing rate (Hz). Only trials with subsequent anticlockwise reaching 
movements are included in the middle and right subpanels. Red vertical lines 
indicate object onset. Firing rate normalized by s.d. c, d, Position of the actuator 
actively controlled (c) and passively observed (d) by monkey M. SNR, signal-to- 
noise ratio. 
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In BCWOH, task requirements were eased: the object selection 
period was reduced to 300-500 ms and monkeys were allowed to 
overstay at an incorrect object. The performance of monkey M, mea- 
sured as the number of rewards per minute, steadily improved from 
1.021+0.007 to 2.962+0.005 (mean+s.em.; Fig. 2d). Similar 
improvements were observed for hand control and BCWH (Fig. 2d, 
inset). The average frequency of actuator displacements, calculated 
from power spectra, was correlated with the improvement in perform- 
ance during BCWOH (R* = 0.16 for the horizontal (x) coordinate and 
R? = 0.26 for the vertical (y) coordinate; P< 0.001, F-test), which 
indicated that the monkey modulated its brain activity to scan the 
targets faster. This behaviour was not random, as the exploration 
interval for NAT (3,620 + 350 ms; mean + s.e.m.) was significantly 
shorter (P<0.02, Wilcoxon rank-sum test) than for UAT 
(4,270 + 310 ms). The exploration of RAT (2,255 + 94ms) was the 
shortest owing to the reduced selection period. For monkey N, 
BCWOH performance (2.084 + 0.085 rewards per minute) did not 
change within sessions, and the differences in exploration intervals 
were not significant. 

In agreement with others*®°°, we observed that M1 neurons repre- 
sented the movements of the actuator even when it was passively 
observed by the monkey (Supplementary Fig. 7). Actuator movements 
(task V) replayed for the monkeys could be reconstructed from M1 
activity, using a separately trained decoder (Fig. 4d), with accuracy 
similar to that in reconstructions made for hand control (Fig. 4c). 
M1 representation of the passively viewed actuator is consistent with 
our suggestion that a neuroprosthetic limb might become incorpo- 
rated in brain circuitry’. 

Our BMBI demonstrated direct bidirectional communication 
between a primate brain and an external actuator. Because both the 
afferent and efferent channels bypassed the subject’s body, we propose 
that BMBIs can effectively liberate a brain from the physical con- 
straints of the body. Accordingly, future BMBIs may not be limited 
to limb prostheses but may include devices designed for reciprocal 
communication among neural structures and with a variety of external 
actuators. 
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METHODS SUMMARY 


All animal procedures were performed in accordance with the National Research 
Council’s Guide for the Care and Use of Laboratory Animals and were approved 
by the Duke University Institutional Animal Care and Use Committee. Two 
rhesus monkeys were implanted with microwire arrays in both brain hemispheres. 
These implants were used for both recordings and ICMS (symmetric, biphasic, 
charge-balanced pulse trains; 100-200 1s, 120-200 }1A). Monkeys manipulated a 
joystick to cause an actuator (computer cursor or a virtual-reality arm) to reach 
towards up to three objects displayed on a computer monitor. The task required 
searching for the single object with particular artificial tactile properties. Objects 
consisted of a central response zone and a peripheral feedback zone. Artificial 
tactile feedback was delivered when the actuator entered the feedback zone and 
continued in the response zone. Holding the actuator over the correct object for 
0.8-1.3 s produced a reward (fruit juice). Holding the actuator over an incorrect 
object cancelled the trial. In brain control trials, the actuator was controlled by 
cortical ensemble activity decoded using an unscented Kalman filter’’. An inter- 
leaved scheme of alternating recording and stimulation subintervals (50 ms each, 
50% duty cycle) was implemented to achieve concurrent afferent and efferent 
operations. In all offline analyses, ICMS periods were excluded from calculations 
of neuronal firing rates. The virtual-reality arm was animated using 
MOTIONBUILDER (Autodesk). 


Full Methods and any associated references are available in the online version of 
the paper at www.nature.com/nature. 
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METHODS 


Subjects and implants. Two adult rhesus macaque monkeys (Macaca mulatta) 
participated in this study. Each monkey was implanted with four 96-microwire 
arrays constructed of stainless steel 304. Each hemisphere received two arrays: one 
in the upper-limb representation area and one in the lower-limb representation 
area. These arrays sampled neurons in both M1 and S1. We used recordings from 
the right-hemisphere arm arrays in each monkey, because each manipulated the 
joystick with its left hand. Within each array, microwires were grouped in two four- 
by-four, uniformly spaced grids each consisting of 16 electrode triplets. The sepa- 
ration between electrode triplets was 1 mm. The electrodes in each triplet had three 
different lengths, increasing in 300-1m steps. The penetration depth of each triplet 
was adjusted with a miniature screw. After adjustments during the month following 
the implantation surgery, the depth of the triplets was fixed. The longest electrode in 
each triplet penetrated to a depth of 2mm as measured from the cortical surface. 

Tasks. The monkeys were trained to manipulate a computer cursor or a virtual- 
reality arm and to reach, using this actuator, towards objects displayed on a 
computer monitor. The objects were visually identical, but had different tactile 
properties as conveyed by ICMS of S1. In hand control, each trial commenced 
when the monkey held the joystick with their working hand. Then a target 
appeared in the centre of the screen. The monkey had to hold the actuator within 
that centre target for a random hold time uniformly drawn from the interval 0-2 s. 
After this, the central target disappeared and was replaced by a set of virtual objects 
radially arranged about the centre of the screen. Each of these consisted of a central 
response zone and a peripheral feedback zone, distinguished by their shading 
(Supplementary Fig. 1c). Tactile feedback was delivered in the feedback zone or 
the corresponding response zone. For monkey M, the radius of the response zone 
varied from 1.5 to 4.0cm and the radius of the feedback zone varied from 4.5 to 
7.25 cm, across all tasks and sessions. For monkey N, the radius of the response 
zone varied from 1.5 to 4.5 cm and the radius of the feedback zone varied from 4.75 
to 9.5 cm, across all tasks and sessions. A trial was concluded when the monkey 
placed the actuator within the response zone for a hold interval (800-1,300 ms for 
hand control, depending on the session; 300-500 ms for brain control) or the 
monkey released the joystick handle (in hand control trials). The next trial com- 
menced after an intertrial interval of 500 ms. 

The sequence of events was the same during brain control trials. In some brain 
control sessions, the joystick was removed from the behavioural set-up. For these, 
each new trial commenced following the previous intertrial interval without the 
requirement for the monkey to hold the joystick. In tasks I-III, monkeys chose 
from a set of two objects. In task I, the monkeys had to choose between RAT and 
NAT for fixed object locations. In task II, RAT and NAT were presented on the 
screen at different angular locations in each trial. In task III, object number and 
spatial arrangement were the same as in task II, but RAT and UAT were used. In 
task IV, three objects were used (RAT, UAT and NAT) and their arrangement on 
the screen varied from trial to trial. Finally, in task V, the virtual-reality monkey 
arm replaced the computer cursor. 
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Psychometric measurements. Psychometric measurements determined the 
minimum ICMS amplitude that the monkeys could discriminate (Supplemen- 
tary Fig. 2b). In these measurements, the ICMS amplitude was different in every 
trial. In each psychometric session, a range of amplitudes was selected such that 
about half were in a range clearly above the monkeys’ threshold for discrimination 
and half were in a range of unknown discriminability. 

Catch trials. In some sessions, a small percentage of trials (typically 1%) were 
designated as catch trials. In these trials, the microstimulator delivered pulse trains 
with zero amplitude, but all other aspects of the behavioural task remained the 
same. This allowed us to confirm that there were no unintentional sources of 
information that the monkeys could use to perform the tasks. 

Algorithms. An Nth-order unscented Kalman filter?* (UKF) was used for brain 
control predictions. Up to a tenth-order UKF was used in some sessions, but in 
most sessions we found that the third-order UKF was sufficient. The filter para- 
meters were fitted on the basis of either the hand movements of the monkeys while 
they performed the task using a joystick or on passive observation of actuator 
movements while the monkeys’ arms were restrained. 

ICMS. Symmetric, biphasic, charge-balanced pulse trains were delivered in a 
bipolar fashion across pairs of microwires. The channels selected had clear sensory 
receptive fields in the upper limb (monkey M: two pairs of microwires with 
synchronous pulse trains) or lower limb (monkey N: one pair of microwires). 
For monkey M, the anodic and cathodic phases of stimulation had a pulse width 
of 105 ts; for monkey N, the pulse width was 200 pts. The anodic and cathodic 
phases of the stimulation waveforms were separated by 25 1s. 

Interleaved ICMS and recordings. We implemented an interleaved scheme of 
alternating recording and stimulation intervals (Supplementary Fig. 2a). Our BMI 
had a 10-Hz update rate. That is, 100 ms of past neural data were used to make 
predictions about the desired state of the actuator. We broke up each 100-ms 
interval into two 50-ms subintervals. In the first subinterval (Rec), neural activity 
was recorded as usual and the measured spike count was used to estimate the firing 
rate for the whole 100-ms interval. The second subinterval (Stim) was reserved 
exclusively for delivering ICMS; all spiking activity occurring in this subinterval 
was discarded. Whenever the actuator was in contact with a virtual object at the 
start of a Stim interval, an ICMS pulse train was delivered. For RAT, nine pulses of 
ICMS were delivered; for UAT, 18 pulses of CMS were delivered; and for NAT, no 
pulses of ICMS were delivered. The neural activity in the Stim interval was dis- 
carded even for NAT, so that there would be no bias induced by ICMS-occluded 
neural data. 

Virtual-reality monkey arm. In task V, we introduced a novel, brain-controlled 
virtual-reality arm with realistic kinematic movements and spatial interactions. 
The control loop rate was 50 Hz, with visual refreshing at 30 Hz. The arm model 
was designed to depict a rhesus macaque. We presented a first-person perspective 
of the virtual-reality arm to the monkey, who controlled the position of the hand. 
Arm posture was controlled using a mixture of direct control of end effectors and 
inverse kinematics, constrained by the physical interdependencies of the joints. 
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Fitts’ law describes the fundamental trade-off between movement accuracy and speed: 
it states that the duration of reaching movements is a function of target size (TS) and 
distance. While Fitts’ law has been extensively studied in ergonomics and has guided the 
design of human-computer interfaces, there have been few studies on its neuronal cor 
relates. To elucidate sensorimotor cortical activity underlying Fitts’ law, we implanted two 
monkeys with multielectrode arrays in the primary motor (M1) and primary somatosensory 
(S1) cortices. The monkeys performed reaches with a joystick-controlled cursor toward tar 
gets of different size. The reaction time (RT), movement time, and movement velocity 
changed with TS, and M1 and S1 activity reflected these changes. Moreover, modifica- 
tions of cortical activity could not be explained by changes of movement parameters alone, 
but required TS as an additional parameter. Neuronal representation of TS was especially 
prominent during the early RT period where it influenced the slope of the firing rate rise 
preceding movement initiation. During the movement period, cortical activity was corre- 
lated with movement velocity. Neural decoders were applied to simultaneously decode 
TS and motor parameters from cortical modulations. We suggest that sensorimotor cortex 
activity reflects the characteristics of both the movement and the target. Classifiers that 
extract these parameters from cortical ensembles could improve neuroprosthetic control. 


Keywords: motor cortex, somatosensory cortex, sensorimotor transformation, neurophysiology, Fitts’ law, decision 


making, brain-machine interface, neuroprosthetics 


INTRODUCTION 

The relationship between movement speed and accuracy, first 
reported by Shannon and Weaver (1949) and Fitts (1954), is com- 
monly referred to as Fitts’ law and is formulated as a dependency 
of movement time (MT) on target size (TS) and distance to the 
target. In information theory and communication fields, this law 
provides a means to quantify the information capacity of a motor 
system. Fitts’ law has been extensively studied and confirmed to 
hold for upper limb (Leisman, 1989; Bootsma et al., 1994) and 
head movements (Jagacinski and Monk, 1985), both in adults and 
children (Kerr, 1975; Hay, 1981). Fitts’ law disturbance has been 
described for neural diseases such as developmental coordination 
disorder (Maruff et al., 1999; Wilson et al., 2001) and for Parkin- 
son’s disease (Mazzoni et al., 2007, cf. Weiss et al., 1996). There 
are conditions for which additional factors have to be accounted 
for, including eye movements (Chi and Lin, 1997) and ballistic 
arm movements (Hoffman, 1991). For the past 30 years, issues of 
computer interface design have driven much interest in Fitts’ law, 
helping to improve pointing time through improved screen lay- 
out and menu design (Gillan et al., 1992). More recently, it has 
been shown that Fitts’ law accurately describes the MTs of a one 
dimensional cursor in an electroencephalographically (EEG) dri- 
ven brain—machine interface (BMI), both in normal subjects and 
in patients with amyotrophic lateral sclerosis and spinal muscular 
atrophy (Felton et al., 2009). 


From a neurophysiological perspective, Fitts’ law describes 
a sensorimotor transformation in which sensory evidence is 
detected by the visual system, analyzed by brain visuomotor net- 
works, and is eventually converted into movements toward a target 
(Kalaska et al., 1997). The neuronal mechanisms of sensorimo- 
tor transformations and decision making have been extensively 
studied (reviewed in Romo and Salinas, 2001; Gold and Shadlen, 
2007; Hoshi and Tanji, 2007; Lalazar and Vaadia, 2008; Andersen 
and Cui, 2009; Paz and Vaadia, 2009; Cisek and Kalaska, 2010). 
These studies have suggested that there is no strict segregation 
between processing in sensory and motor areas: the cortical areas 
involved in movement execution, such as motor and premotor cor- 
tex, also represent sensory signals and participate in sensorimotor 
transformations (Alexander and Crutcher, 1990; Crammond and 
Kalaska, 1994; Zhang et al., 1997; Wise et al., 1998; Kakei et al., 
1999; Paz etal., 2003). Furthermore, motor areas appear to sequen- 
tially represent a variety of parameters: abstract parameters related 
to task rules, orientation of spatial attention (Lebedev and Wise, 
2002), motor planning and anticipation (Vaadia et al., 1988; Wise 
et al., 1996), and characteristics of limb kinematics and kinetics 
(Todorov and Jordan, 2002; Cisek et al., 2003; Sergio et al., 2005; 
Xiao et al., 2006; Fagg et al., 2009). According to the evidence- 
accumulation model, distinct motor program parameters, such as 
reaction time (RT), and movement velocity, emerge as a result of 
evidence compiled from sensory information (Schall, 2003; Gold 
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and Shadlen, 2007; Kable and Glimcher, 2009). Notwithstanding 
this previous work on sensorimotor transformations, the specific 
dependencies described by Fitts’ law and their underlying neuronal 
mechanisms have not been investigated using neurophysiological 
approaches. 

To explore the neural correlates of Fitts’ law, we conducted 
experiments in two rhesus macaques implanted with multielec- 
trode arrays in primary motor (M1) and primary sensory (S1) 
cortex. Monkeys performed a reaching task in which TS was 
variable. Our results elucidated the representation of TS in sen- 
sorimotor cortex, which was conjoint to the representation of 
movement direction and speed. 


MATERIALS AND METHODS 

IMPLANTS AND RECORDINGS 

All studies were conducted with approved protocols from the Duke 
University Institutional Animal Care and Use Committee and were 
in accordance with the NIH guidelines for the Care and Use of 
Laboratory Animals. 

Two rhesus monkeys (one male and one female, monkeys M 
and N, respectively) were chronically implanted with multielec- 
trode arrays in M1 and S1 of both right and left hemispheres using 
previously described surgical methods (Nicolelis et al., 2003). 
Within each hemisphere, two 96 channel microelectrode arrays 
were placed in cortical areas corresponding to cortical represen- 
tations of arm and leg (Figure 1B). Each array consisted of two 
4x 4 grids of independently movable electrode triplets. Each of 
the triplets was comprised of electrodes of different lengths, in 
300 .m intervals, which allowed us to sample neuronal activity 
from different depths in the cortical tissue. For the purpose of the 
study, neural activity was recorded in the arm representation area 
of right hemisphere M1 (in both monkeys) and S1 (only in mon- 
key M). Recorded signals were amplified, digitized, and filtered by 
a multichannel recording system (Plexon Inc., Dallas, TX, USA). 
Neuronal spikes were sorted on-line using waveform template 
matching and thresholding features built into the spike-sorting 
software. 


BEHAVIORAL TASK 
Each monkey was trained to perform a two dimensional reaching 
task using a hand-held joystick that controlled the position of a 
computer cursor. The monkeys made center-out movements to 
peripheral targets. This experimental protocol matched the design 
implemented in a previous human study of Fitts’ law for pointing 
movements (Smyrnis et al., 2000). In our experiments, the joy- 
stick was at the monkey’s waist level on the side of the working 
hand. The monkeys used their left hands to hold the joystick. The 
left hand was chosen because the quality of neuronal recordings 
was better in the right hemisphere in each monkey. X (left-right) 
and Y (forward—backward) positions of the joystick was trans- 
lated to the X (left-right) and Y (up-down) positions of the 
cursor on the display screen (Figure 1A). The display screen was 
placed 45cm from the monkeys’ eyes, and the cursor diameter 
was 0.5 cm. 

To initiate a trial, the monkey positioned its hand on the joy- 
stick. The trial was canceled if the monkey broke hand contact 
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FIGURE 1 | Implantation and experimental protocol. (A) Rhesus 
monkeys controlled the location of a cursor on a display screen by moving a 
joystick with their left hand. Joystick kinematics as well as the neural 
activity were recorded and analyzed offline. (B) 4 x 4 Grids of 16 electrode 
riplets were implanted bilaterally in M1 and S1 arm and leg regions, 
however only the right hemisphere arm region of M1 and S1 was recorded 
rom in this study. (C) For each trial, the cursor was to move along the radial 
origin-to-target axis (X’) toward one of four potential target locations. (D) 
Left to right-typical trial begins with cursor moved within the target at 
center of screen. After hold period, penalty ring, and target arc appear. The 
cursor is then moved radially through the target arc to receive a reward. (E) 
Three potential target sizes are shown with respect to the cursor, for size 
reference. (F) An example of a single trial movement trace is shown. Target 
onset (TO) and movement onset (MO) are denoted on time axis. The 
approach epoch that was used in later analysis spanned from movement 
onset to target acquisition. 


with the joystick. Once the monkeys touched the joystick, a com- 
puter cursor was shown on the screen and a 3-cm diameter circle 
appeared at the center of the screen. The monkey moved the cursor 
inside that circle, and held it there for a random interval between 
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800 and 1500 ms (Figure 1D). After this hold period, the central 
circle disappeared and the monkey was required to move the cursor 
to a peripheral target that appeared on the screen simultaneously 
with the central circle disappearance. The peripheral target was 
a thickened arc of either 8, 15, or 22° on a thin boundary circle 
aligned on the center of the screen (Figure 1E). A juice reward was 
immediately delivered when the cursor crossed the target from the 
inside out (Figure 1D, right). Movement of the cursor out of the 
boundary circle that did not cross the target resulted in termina- 
tion of the trial without reward and a 500-ms timeout period. The 
targets appeared at angles 45, 135, 225, and 315” relative to the 
center of the screen (Figure 1C). The monkey had 5 s to complete 
each trial. This experiment was repeated over three daily recording 
sessions in monkey M and four sessions in monkey N. 


ANALYSIS OF MOVEMENTS 
To calculate movement onset and velocity for different target posi- 
tions, single trial records of cursor position were analyzed using 
a coordinate transfer where a new coordinate system (X’, Y’) was 
obtained by rotating the (X, Y) axes so that the new X’ axis coin- 
cided with the radial axis from the screen center to the target, and 
Y’ axis was orthogonal to X’ (Figure 1C). The time of movement 
onset was detected by identifying the first co-occurrence of both 
the X’ velocity and acceleration surpassing an empirically deter- 
mined threshold of 0.25-Velmax and 0.25-Accelmax, where Velmax 
and Accelmax are the respective maximum values reached on a 
given single trial. This algorithm was verified by visual inspection 
of the accurate identification of movement onset on greater than 
95% of tested trials. To detect movement onset in the trials where 
the co-occurrence of velocity and acceleration increases were not 
found, the first instance of one of the two events was selected. Out- 
lier trials with unusual movement patterns (i.e., return to the cen- 
ter before reaching the target) were excluded from analysis. We call 
the period between the movement onset and the passing of the cur- 
sor into the target the approach epoch (Figure 1F). The mean veloc- 
ity was calculated within this epoch for each trial. RT was defined 
as the time elapsed from target appearance to movement onset. 
Trial difficulty was quantified as a function of target width, for- 
malized in terms of the index of difficulty (ID) as described in Eq. 
1 (Fitts and Peterson, 1964): 


ID = log, (2D/W) (1) 
where D is the distance from screen center to target and W is the 


length of the target arc (8°—0.56, 15°-1.05, 22°-1.54cm). D was 
fixed at 4cm. 


ANALYSIS OF NEURONAL ACTIVITY 

Neural activity was first analyzed using conventional peri-event 
time histograms (PETHs; Awiszus, 1997), either aligned on target 
onset or on movement onset. Recorded action potential events 
were counted in bins of 25 or 50 ms width. The smaller bin width 
was used for computation of slopes and ensemble-average mean 
firing rate (MFR) to improve temporal resolution. PETHs for a 
single neuron were calculated for each trial and then were aver- 
aged across trials for each combination of movement direction 
(four possible) and TS (three possible). This average modulation 


profile for each neuron was normalized by subtracting the mean 
bin count and dividing by the SD of the cell’s bin count — the 
values calculated for raw spike trains prior to any PETH calcula- 
tions. With this normalization, PETH expresses the event-related 
modulations as a fraction of the overall modulations, or statis- 
tically, the z-score. The preferred direction for each neuron was 
determined by finding the direction with maximum MER in the 
750-ms window following target onset. 

Trials from each session were subdivided into 12 groups (by 
movement direction and TS) and subsequent analysis was per- 
formed separately for these groups. Trials were further categorized 
into two categories by the length of RT relative to the median values 
for that session. Trials with RT below the median were character- 
ized as short RT, and above the median as long RT. Similarly, trials 
were categorized by approach velocity as slow and fast velocity tri- 
als. The corresponding neuronal activity profile was computed for 
each selected subset of trials as a way to isolate the neural correlates 
of specific differences in behavior. 

The typical response profile for both M1 and $1 movement- 
related neurons was a rapid increase in firing rate (FR) beginning 
approximately 50-100 ms after target onset (Figure 4). Maximum 
FR was typically reached near the time of movement onset. Neural 
activity between the target onset and movement initiation (the RT 
period) was examined for modulation with TS. We found that the 
FR slope differed with TS during the RT period. The FR slope for 
a given trial was determined within a fixed interval during the RT 
period, 50-300 ms after target onset, where we performed a linear 
regression across binned (25 ms bin width) spike activity. Mean 
slopes were calculated by averaging across all trials in a specified 
experimental condition (e.g., direction, RT group). Mean slopes 
for each condition were calculated for individual neurons, followed 
by an ensemble-level analysis. 

In addition to categorical statistical analyses, we treated RT 
and TS as continuous, rather than categorical values. To study the 
contribution of RT and TS to slope modulation, we performed a 
multiple linear regression analysis where the FR slope of each cell 
was expressed as a linear function of the RT and TS (Eq. 2). 


Slope = Wer + Wrs + C (2) 


Weights Wr and Wvs expressed the contribution of RT and 
TS, respectively. Wer and Ws were determined for each neu- 
ron along with the 95% confidence interval (CI) of their values 
(Ashe and Georgopoulos, 1994). Each CI was generated using the 
student’s f statistic with a = 0.05. A neuron was said to enact a 
significant modulation for RT, TS, or both if the corresponding CI 
did not contain zero. The subset of neurons significantly correlated 
to RT or TS were found using this criterion. 

For the peri-movement epoch (PME), we found that MFR was 
sensitive to TS and movement velocity. For analyses of the rela- 
tive magnitude of neural activity prior to and during movement, 
PETHs were aligned on movement onset and MER was calculated 
in the PME, defined to be the interval from 50 ms before to 50 ms 
after movement onset. MFR modulation could be broadly seen 
with comparisons between discrete categories of velocities; how- 
ever a more nuanced analysis once again required a multiple linear 
regression of continuous variables. Similar to Eq. 2, MFR during 
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the PME was fit as a linear function of velocity and TS: 
MFR = Wye + Wrs + C (3) 


Where Vel is movement velocity. Weights Wy and Wrs 
expressed the contribution of velocity and TS on MER. Statis- 
tics of single cells and across the population were computed using 
the same technique as for Eq. 2. 


NEURAL REPRESENTATION OF TASK AND BEHAVIORAL PARAMETERS 
AS A FUNCTION OF TIME 

Neuronal representation of different task and behavioral para- 
meters was quantified using trial-by-trial correlation analyses 
between the variations of neuronal rates and parameters in ques- 
tion (RT, Velocity, and TS). More specifically, we conducted mul- 
tiple linear regression analysis in which a 50-ms window was slid 
along the task interval (Figure 6A). FR was calculated within the 
window for each trial and fit with a linear function of either RT 
and TS (Eq. 4) or mean velocity and TS (Eq. 5): 


FR= Wert Wrs + C (4) 
FR = Wa + Wrs + C (5) 


We found that FR was correlated with RT mostly after target 
appearance and with Vel mostly around movement onset. There- 
fore neural data was aligned on target onset for regression in Eq. 
4 and on movement onset for regression in Eq. 5. The magnitude 
and sign of each coefficient W provided information regarding the 
effect that the corresponding parameter had on FR of a single cell 
at a given time. Coefficients for individual neurons were averaged 
across neurons at each time step. 


NEURAL DECODERS 

To evaluate the accuracy with which task and behavioral para- 
meters can be extracted from ensemble activity, we used a neural 
classifier and linear predictor that decoded kinematic and tar- 
get parameters. Linear discriminant analysis (LDA; Fisher, 1936) 
was performed using a 100-ms wide window slid at 25 ms time 
steps. MFR was measured within this window for each neuron 
in the population. The vector of individual neurons’ MFRs was 
the input to the decoder. We used 60% of trials as sample data 
for the decoder, and 40% of trials as test data. LDA predicted 
categorized variables from neuronal ensemble activity. For each 
predicted parameter, we used three categories of either TS (8°, 15°, 
22°), RT, or velocity (lower 1/3, middle 1/3, upper 1/3 RT, or veloc- 
ity). The performance of the classifier was measured as the fraction 
correct prediction divided by the chance level fraction correct and 
was found at each time step of the sliding window to provide a 
temporal profile (Figure 8). Fraction correct for a specified sliding 
window position was averaged between analyses where the sam- 
ple data were drawn from the beginning, middle, and end of the 
session. 

The Weiner filter (Wiener, 1949; Haykin, 2002) was used offline 
to make predictions of cursor kinematics using a short history 
of neural activity (Wessberg et al., 2000; Lebedev et al., 2005). 
This analysis tested the influence of TS, RT, and Vel on cursor 
position extracted from cortical activity (Figure 7). The Wiener 


filter used in this study had six taps of 100 ms with ridge regres- 
sion for regularization (Grandvalet, 1998). The filter performance 
was evaluated using both signal-to-noise ratio (SNR; Fitzsimmons 
et al., 2009; Li et al., 2009) and coefficient of determination (r?; 
Wessberg et al., 2000; Lebedev et al., 2005). 


RESULTS 

Data were collected from four daily recording sessions in monkey 
N (2126 trials) and from three sessions in monkey M (1305 trials). 
Neural activity was recorded from 64 to 69 M1 neurons (range for 
different recording sessions) in monkey N, and from 92 to 111 M1 
neurons and 83—91 S1 neurons in monkey M. 


INFLUENCE OF TARGET SIZE ON REACTION TIME AND MOVEMENT 
PARAMETERS 

Reaction time and MT were affected by TS in a manner consistent 
with Fitts’ law in both monkeys. Additionally, these changes were 
dependent on movement direction. RT, determined by subtracting 
target onset from movement onset, clearly decreased with increases 
in TS. The distributions of RTs for each of the three TS and each 
of four movement directions are shown in Figures 2A—C. Because 
these distributions were non-Gaussian in shape, the Kruskal— 
Wallis test, a non-parametric analog of ANOVA, was used to 
compare RTs for different conditions. This analysis showed that 
the TS significantly influenced the distribution of RTs (p < 0.001 
for all movement directions with the exception of one for monkey 
M with p <0.02; see Table 1). The TS effect on RT was espe- 
cially pronounced for upward-rightward movements in monkey 
N (Figure 2A) and downward-—leftward movements in monkey 
M (Figure 2B).The effect was the smallest for upward—leftward 
movements in both monkeys. That direction also had the overall 
shortest RT, which suggests that those particular movements were 
relatively easy for the monkeys to prepare and perform. 

The target approach time (TAT; i.e., the time from movement 
onset until the time of target acquisition) also depended on the 
TS (p <0.01, Kruskal-Wallis test) for each movement direction 
in both monkeys (Table 1). TAT was shorter for larger targets 
and longer for smaller targets. As with RTs, the TS effect was 
more pronounced for particular directions and less for others. 
The particularly vigorous (i.e., characterized by higher velocity) 
upward-—leftward movements depended on the TS less than move- 
ments in other directions. This can be seen from Figures 3C,D 
which presents the distributions of average approach velocities, 
that is, the inverse of TAT. The effect of TS on the RT and movement 
kinematics is also clear from average movement traces showing 
time dependent changes in cursor position measured with respect 
to the radial axis from the center to the target, termed X’ axis 
(Figures 3A,B). The average cursor traces for the smallest tar- 
get is clearly shifted to the right for both monkeys, reflecting TS 
dependent differences in RTs. Differences in movement velocity 
are manifested as differences in the slopes and are especially clear 
for the initial portion of movement. The initial velocity was slow- 
est for the smallest targets (black traces) and fastest for the largest 
(red traces). 

Since both RT and TAT elongated with smaller targets, MT, 
defined as the time from initial appearance of the target to when 
the target was entered, elongated as well. The dependency of RT on 
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FIGURE 2 | Distribution of reaction times was computed for each 
movement direction by monkey N (A) and monkey M (B) with 
probability “P” shown as a function of RT. A Kruskal-Wallis test was 
performed for each direction to determine significance of target size on 
distribution of reaction times (see Table 1). (C) Reaction time for trials of the 
three different indices of difficulty (ID) was fit with linear function and tested 


A Monkey N B Monkey M 
0.4 0.25 
5 pa000! p<0.001 p<0.02 p<0.001 
P 
pee 45° 135° 45° 
0 0 0 
si 225° \ 315 _ a 315 
0.001 
P p< p<0.001 p p<0.001 p<0.001 
0 O35 2 0 05 2 
Reaction time (s) Reaction time (s) 
Cc Target Size D 
0.6 2.4 
a a) 
3s S p<0.01 
£ £ 21.35 bits/s 
E p<0.01 = 
5 i § 
$ 7 wee MonkeyM_ ¢€ 
6 al v 
© = — -MonkeyN 2 < <0.01 
p<0.01 y 3 Pp 
=” e, ~ WF 301 bits/s 
"2.2 ID 4 2.2 iD 4 


for significance using F test. Means for each ID plotted as filled circles 
(monkey M) and open circles (monkey N). The target size of the trial is 
denoted by colors specified below (A,B). (D) Movement time for the three ID 
conditions. A regression line was used to fit all trials and the subsequent 
inverse of slope yields index of performance (IP) in bits per second. 
Significance was tested in same way as in (C). All error bars indicate SE. 


Table 1 | Kruskal-Wallis analysis of the effect of target size on reaction time and target approach time. 


Direction Monkey N Monkey M 
Target size x reaction time Target size x velocity, TAT Target size x reaction time Target size x velocity, TAT 
45° p<0.001 p<0.001 p<0.001 p<0.001 
135° p<0.001 p<0.001 p<0.02 p<0.01 
225° p<0.001 p<0.001 p<0.001 p<0.001 
315° p<0.001 p<0.001 p <0.001 p<0.001 


p-values determined for target size effect on both reaction time and velocity (and therefore target approach time), in each of the four movement directions. 


ID is represented in Figure 2C. Average MT tended to be longer 
for higher ID trials (F test, p< 0.01), following Fitts’ law, with 
an index of performance (IP) of 21.35b/s for monkey M and 
8.21 b/s for monkey N (Figure 2D). The IP was determined by 
linear regression following the original method of Fitts (1954). 
Taken together, these results indicate that monkeys’ reaching 
movements obeyed Fitts’ law. Small targets were associated with 
elongated RT, slower movement velocity, and an overall lengthen- 
ing of MT in a manner that is consistent with previous Fitts’ law 
studies (Bohan et al., 2003; Munro et al., 2007; Boyd et al., 2009). 


NEURONAL REPRESENTATION OF REACTION TIME AND TARGET SIZE 
Given that TS affected RTs, TATs, and movement profiles, we 
expected and observed that these changes in movement kinematics 


were reflected in cortical modulations. However, modifications in 
M1 and S1 neuronal activity were not mere reflections of changes 
in movement patterns. Rather, they were better explained by a 
combination of factors that included behavioral parameters and 
TS than by movement alone. In other words, neuronal modula- 
tions in sensorimotor cortex did not simply reflect the character- 
istics of movement, but additionally depended on the properties 
of reach targets. 

Neuronal modulations can be clearly seen in PETHs separated 
for different RTs and TS (Figure 4). These PETHs are shown 
as color plots for the M1 and S1 subpopulations of neurons 
(Figures 4D-I). Population-average PETHs were computed sep- 
arately for M1 (both monkeys; Figures 4A,B) and S1 (monkey 
M only; Figure 4C). Both individual-neuron and average PETHs 
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FIGURE 3 | Reach kinematics reflect differences in target size. (A,B) 
Averaged position traces of monkey N and monkey M along the X’ axis from 
0 (the origin) to 4 (the target, denoted by dashed line). The target size of the 
trial is denoted by colors specified in (A). (C,D) Distribution of mean approach 
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velocity for each of the four movement directions with probability (P) shown 
as a function of mean approach velocity. For each direction, a Kruskal-Wallis 
test was performed to evaluate the effect of target size; p-values shown for 
each direction separately (see Table 1). 


were clearly different for different RTs. Shorter RTs corresponded 
with PETHs with steeper modulation slopes. To examine the 
effects of both RT and TS, a two-factor analysis was conducted. 
Behavioral trials were categorized into (1) short and long RT 
groups and (2) small (8°), intermediate (15°), and large (22°) TS 
groups. RTs lower than the median RT were classified as short, 
and RTs higher than the median as long. Additionally, preferred 
directions (movement direction with maximum neuronal rate) 
and least preferred directions (minimum-rate direction) were ana- 
lyzed separately for each neuron. In Figure 4, panels for short vs. 
long RTs, different TS and preferred vs. least preferred directions 
are labeled accordingly. Analyses of these groupings suggested that 
PETH slopes increased with shorter RTs and additionally increased 
with larger targets. PETH slopes for individual neurons and pop- 
ulations were calculated using linear regression of PETH values 
vs. time in the interval from 50 to 300 ms after target onset (see 
Materials and Methods). Average population PETH separated into 
most and least preferred direction are shown in Figures 4A—C. 
They indicate a tendency for the slopes to depend on both RT and 
TS for both preferred and least preferred directions. For both pre- 
ferred and least preferred direction, average slopes increased with 
shorter RTs and larger TSs. 

An analysis of PETH slopes for single neurons revealed the 
same tendency as for the slopes of population PETHs (Table 2). 
For the preferred direction, short RT trials had, on average, steeper 
slopes than long RT trials for M1 (monkey N and M), as well as 
S1 (monkey M) cells (p < 0.0001, paired t-test). The same ten- 
dency was found for the least preferred direction in monkey N 
M1 and monkey M M1 neurons (p < 0.0001), however the effect 


was not significant in the recorded S1 cell population (p > 0.05). 
Furthermore, PETH slopes in individual neurons, both in the pre- 
ferred and the least preferred directions showed TS dependence 
among all trials in monkey N M1, monkey M M1, and monkey 
M S1 (p <0.01). To test significance of this effect, we performed 
both one-way ANOVA (ER slope distribution was nearly normal) 
and the Kruskal—Wallis test. Reported is the larger of the two p- 
values for each analysis. Looking within only long RT trials, the 
effect remained significant in both the preferred and least pre- 
ferred direction in each neuronal population, in both monkeys 
(p < 0.001). In short RT trials, the TS effect was significant in mon- 
key N M1 (p < 0.001) and monkey M S1 (p < 0.01) but only mar- 
ginally significant in monkey M M1 (p=0.12 ANOVA, p < 0.05 
Kruskal—Wallis), possibly because monkeys were less careful to hit 
the target on those trials and their movements were more ballistic, 
which have been shown not to follow Fitts’ law (Juras et al., 2009). 
A greater difference emerged between the small target trials and 
either of the other two larger target trials, both at the behavioral 
and neuronal levels. As such, further slope comparisons were made 
between the small target trials and each of the other two groups 
in a pairwise manner (Table 2). Neural activity of the population 
followed this trend as well (Figures 4D-I). The results were con- 
sistent in both preferred and least preferred directions, in M1 as 
well as S1 in monkey M. 

While the analysis of Figure 4 suggested that PETH slope 
could not be explained solely by changes in RT and additionally 
depended on TS, this demonstration was not sufficient because 
the composition of short RT and long RT groups of trials could be 
different for different TSs. In particular, RTs could be shorter for 
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Table 2 | Effect of reaction time and target size on firing rate slope preceding movement onset. 


Monkey N, M1 cells 


Monkey M, M1 cells 


Monkey M, S11 cells 


Preferred Least Preferred Least Preferred Least 
Long RT 0.114+0.01 0.085+ 0.01 0.232 40.01 0.202 +0.0 0.092 + 0.01 0.096 + 0.01 
Short RT 0.266 + 0.01 0.257+0.01 0.345 + 0.01 0.278 + 0.0 0.143 +0.01 0.112+0.01 
(0.53)** (0.57)** (0.69)** (0.56)** (0.58) ** (0.53)' 
Small 0.058 + 0.01 0.077+0.01 0.260 + 0.01 0.192 + 0.01 0.058 + 0.01 0.077 40.01 
Medium 0.153+ 0.01 0.104+ 0.01 0.297 +0.01 0.263 + 0.0 0.152 +0.01 0.104+ 0.01 
(0.64)*** (0.57)* (0.57)*** (0.60)*** (0.64)*** (0.57)* 
Large 0.143 +0.01 0.132 + 0.01 0.313+ 0.01 0.265 + 0.0 0.143+0.01 0.132 + 0.01 
(0.65)*** (0.61)*** (0.61)*** (0.62)*** (0.65)*** (0.61)*** 


Comparison of firing rate slope during reaction time period for both monkeys in M1 and S71 (monkey M only). Slopes were computed during a fixed window of 


the PETH of neurons with the specified population during movements in either the preferred (left column) or least preferred direction (right column) and averaged 


across cells and multiple sessions. Rows represent the compared groups of reaction times (rows 1, 2) and the groups of target size (rows 3-5). Data shown is mean 


slope + SD. In parentheses is fraction of recorded cells with slope for test condition greater than slope for control condition. Long RT and small targets chosen as the 


control groups, with paired t-test performed to evaluate significant differences between AT or target size groups. 


*p < 0.05, **p < 0.001 short RT vs. long RT, ***p < 0.001 medium or large target vs. small target, ‘not significant. 


large TS than for small TS even after they were separated into short 
RT or long RT groups. To address this possibility, we performed a 
linear regression analysis where the slope was expressed as a linear 
function of both RT and TS (Eq. 2). 

The weights Wrr and Ws (Eq. 2) were calculated for different 
cortical areas illustrated (i.e., the same neuronal populations as in 
Figures 4A-C). Mean Wr in all three cortical areas was negative 
(monkey N M1 —0.026 + 0.002, monkey M M1:—0.046 + 0.003, 
and S1: —0.048 + 0.003; mean + SE), confirming the inverse rela- 
tionship between RT and PETH slope. Conversely, mean Ws, in 
all three cortical areas was positive (monkey N M1: 0.046 + 0.002, 
monkey M M1: 0.011 + 0.002, and $1:0.031 + 0.003), confirming 
a slope increase for larger TS. To estimate the significance of these 
values, single cell coefficient values, as well as their corresponding 
95% CI were determined from regression (see Materials and Meth- 
ods). In both monkeys, M1 cells were identified with significant 
Wer or Ws, although the dominant parameter differed between 
the two monkeys (monkey N: TS 40% of cells, RT 18%; monkey 
M: TS 11% of cells, RT 31%). For S1 neurons the modulation was 
slightly more related to RT than TS (TS 20% of cells, RT 32%). 
Thus both TS and RT contributed to modulation of FR slope dur- 
ing the RT period in M1 and S1 neurons. However, the relative 
effect of size was variable between monkeys. 


NEURONAL REPRESENTATION OF VELOCITY 

The changes in mean approach velocity for different TSs (Figure 3) 
were associated with modification in M1 and S1 activity. In addi- 
tion, we observed a modulatory effect of TS on FR beyond the 
changes in neuronal activity explained solely by kinematics. Sim- 
ilar to the analysis of the effect of RT and TS on the initial 
PETH slope, in this analysis we calculated PETHs for groups of 
trials selected based on the mean velocity during the approach 
epoch categorized as slow or fast. PETHs for both groups, in 
both the preferred and least preferred directions were computed 
(Figure 5). PETHs were centered on movement onset. The main 
effect seen in the average PETHs (Figures 5A—-D) and PETHs 
for individual neurons in the population (Figures 5E-J) was an 


increase in neuronal rates for fast velocity trials compared to slow 
velocity trials. Statistical significance of this effect was confirmed 
by an analysis of peak FR (p < 0.001, paired t-test; Figure 5D). 
Furthermore, the neuronal rates were modulated by the TS in 
such a way that for small targets, FR were reduced for both slow 
and high velocities and preferred and least preferred directions 
(Figures 5A—C). PETH amplitude near the time of movement 
onset was found to be dependent on both the velocity of the trial 
and the TS (Table 3). 

A fixed window was set spanning from 50 ms before to 50 ms 
after movement onset denoted as the (PME; see Materials and 
Methods). The MFR in the PME was larger for fast velocity tri- 
als than for slow trials in all cortical areas and monkeys, both in 
the preferred and least preferred directions (p < 0.001, one-way 
ANOVA). In addition to velocity, TS modulated FR throughout 
the trial (Figures 4A—C). Our data confirmed that the TS mod- 
ulated MFR during the PME during movement in the preferred 
direction of M1 and S1 neurons (p < 0.02), and the effect was 
even stronger during the slow trials (p < 0.001). The effect of TS 
on MFR was present in fast trials, but was only seen in M1 of mon- 
key N and only in the preferred direction. Least preferred direction 
modulation was much weaker or was not present in all three cell 
populations recorded. Thus, the MFR was modulated by both TS 
and velocity. 

As velocity and TS were not independent, we next performed a 
linear regression analysis to elucidate their relative contributions. 
The MER during the PME was fit by a linear function of velocity 
and TS (Eq. 3). Weights Wy, and Ws were determined for each 
neuron as well as their 95% CI. FR was found to be more strongly 
influenced by velocity than by TS in each case. In monkey N M1, 
Wve was larger than Wrs [Wye] = 0.019 + 0.003 (mean + SE) 
vs. Wrs = 0.009 + 0.001, p < 0.05, calculated for all cells, not 
just those with significant values], and the fraction of recorded 
neurons reflecting velocity (44.5%) was much higher than the 
fraction reflecting TS (10.9%). Significance was determined using 
the same procedure as for the regression in Eq. 2. In monkey M 
M1, the effect was similar in that the MFR near movement onset 
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FIGURE 5 | Effect of velocity on firing rate profiles. The normalized firing 
rate was computed during the 1-s interval surrounding movement onset from 
three subpopulations of neurons: (A) M1 neurons in monkey N, (B) M1 
neurons in Monkey M, (C) S1 neurons in Monkey M. In each panel, the left 
column represents averaged, normalized FR for movements in each cell's 
preferred direction, right column the least preferred direction. The top row is 
averaged over all trials slower than the median approach epoch velocity and 


E Pref Least Pref Least Pref Least 
H Fast Velocity Trials 

8° 

15° 

22° 


Movement 
onset 5 


eS 


Normalized Firing Rate 


the bottom row shows only fast trials. Target size specified by line color [see 
legend below (B)]. (D) Same data from (A-C) collapsed into simply a 
comparison of slow vs. fast trial average PSTH for each of the three cell 
groups. Population PSTH for slow (E-G) and fast (H-J) trial averages. In each 
panel: Y axis contains all neurons, X axis represents time aligned on 
movement onset (black bar). Color of pixel represents normalized firing rate 
(z-score, see Materials and Methods). 


was more a function of velocity than TS (Wye = 0.027 + 0.004, 
Ws = —0.009 + 0.002, p < 0.05), and there were more neurons 
reflecting velocity (38.3%) than those reflecting TS (18.9%). Mon- 
key M S1 neurons showed an even stronger contribution for 
velocity (Wye = 0.044 + 0.002, Wts = —0.012 + 0.002, p < 0.05), 
with over 55.1% of neurons reflecting velocity and 14.7% neurons 
reflecting TS. Taken together, FR magnitude in the window sur- 
rounding movement onset was more strongly related to velocity, 
suggesting that amplitude of firing encoded velocity, while the 
slope during the RT period (Figure 4) more strongly encoded 
the TS. 


TIME DEPENDENT MODULATION OF NEURAL ACTIVITY WITH 
MOVEMENT PARAMETERS 

To determine task periods during which neuronal activity reflected 
trial-by-trial variations of RT and velocity, we performed a multi- 
ple linear regression analysis (Eqs 4 and 5). MFR during a sliding 
window was fit to linear functions of RT and TS (Eq. 4) and 
approach velocity and TS (Eq. 5; see Materials and Methods). We 
observed an initial negative correlation of RT with spikes within 
500 ms of target onset in all three cell populations during the 
pre-movement, or RT period (Figure 6B). This result reflected 
earlier, more rapid onset of firing for short RT trials. Interestingly, 
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Table 3 | Effect of trial velocity and target size on mean firing rate in peri-movement epoch. 


Monkey N, M1 cells 


Monkey M, M1 cells Monkey M, S1 cells 


Preferred Least Preferred Least Preferred Least 
Slow .29+ 0.02 0.41+0.02 1.31 + 0.02 0.19+0.02 0.73 + 0.02 0.57+0.01 
Fast 44+ 0.03 0.53 + 0.03 1.53 + 0.03 0.314 0.03 1.25 + 0.02 0.7440.02 
(0.67)* (0.62)* (0.59)* (0.55)* (0.82)* (0.53)* 
Small .04 + 0.03 0.35+0.03 1.34+ 0.04 0.24+ 0.03 0.20 + 0.03 0.56+ 0.03 
Medium 54+ 0.03 0.54 + 0.03 1.44 + 0.04 0.27+0.03 1.03 + 0.03 0.59 + 0.03 
(0.75)*** (0.64) * ** (0.56)** (0.51)' (0.93)*** (0.53)' 
Large 50+ 0.02 0.50+0.03 1.48 + 0.03 0.23 + 0.0.03 0.97+0.01 0.58 + 0.03 
(0.73)*** (0.62)*** (0.56)*** (0.48)" (0.92)*** (0.61) 
ANOVA overall p<0.001 p<0.001 p<0.02 n/s p <0.001 n/s 
ANOVA fast only p<0.001 n/s n/s n/s p<0.001 n/s 
ANOVA slow only p<0.001 p<0.001 p<0.001 n/s p<0.001 n/s 


Mean normalized firing rate, shown as mean + SE in 100 ms window around movement onset varies with both average movement velocity as well as with target 


size. Top two rows: a comparison of slow vs. fast trials. Below the fast trial MFR is the fraction of cells with MFRes: > MF Rew in the 100-ms window. Lower rows: 


MFR during trials with different target sizes. Within each column, fraction of cells with MFRiest > MFRsmav — either the medium or large target size groups — along with 


the denotation of significance. One-way ANOVA performed on MFR with the factor being target size. This test was performed among all trials, only among fast trials, 


and only slow trials, with significance denoted in the corresponding location. 


*o < 0.001 difference from slow trial MFR, **p < 0.05 difference from small target MFR, ***p < 0.001 difference from small target MFR. 


a positive correlation of FR and TS coincided with, or slightly 
led the RT representation. This representation of RT and TS was 
present in both M1 and S1. The same analysis was performed for 
regression of MFR as a function of mean approach velocity and 
TS (Eq. 5). Neural activity aligned on movement onset was binned 
and the coefficients Wye, and Ws were determined. A positive 
correlation between velocity and FR was observed near move- 
ment onset in each cell population, although the effect was more 
subtle in monkey M M1 (Figure 6C). This correlation decreases 
shortly after movement onset to sub-baseline levels, then rises 
again 600-800 ms later. Notably, it was the second correlation peak 
that matched up temporally with the highly significant prediction 
levels computed in LDA from Figures 8D-F. TS correlation was 
less clear, however in monkey N M1 since there were significant 
positive coefficient values corresponding to movement onset in 
this animal. It is likely that the contribution of TS was somewhat 
smeared upon realigning the data on movement rather than target 
onset. In both RT and velocity correlations, there was a specific 
interval during the task in which each parameter was encoded by 
modulations in neural activity. TS correlation with neural activity 
was most clear during the RT period of the movement. 


EXTRACTION OF KINEMATICS AND TARGET SIZE FROM CORTICAL 
ACTIVITY 

The observed differences in cortical activity for slow and fast veloc- 
ities of movements were reproduced by a predictive model that 
extracted cursor position from the recorded population activity 
(Figure 7). M1 and S1 neural activity was combined to gen- 
erate predictions using a Wiener filter (Wiener, 1949) trained 
on the first 50% of a session’s trials. Consistent with previous 
BMI studies (Wessberg et al., 2000; Serruya et al., 2002; Car- 
mena et al., 2003), X and Y position of the cursor were predicted 


with high accuracy (Figures 7A,B; SNR = 4.422 dB, r? = 0.80, X- 
Pos; SNR = 4.798 dB, r? = 0.82, Y-pos). Next, cursor movements 
were predicted for 50% of trials for the two velocity groups, 
averaged, and then compared against the actual averaged trajec- 
tory (Figures 7C,D). There was a clear distinction between the 
fast and slow predicted traces and the Wiener predicted trajecto- 
ries closely matched the actual traces in both monkeys (monkey 
N: 13.98 dB (r7=0.99) fast, 8.54dB (1? =0.95) slow; monkey 
M: 13.73dB (1? =0.99) fast, 13.56dB (1? =0.97) slow). Simi- 
larly, we wondered whether the cursor position could be decoded 
between TS groups. The mean trajectory was computed across a 
single representative session for each monkey for trials of each 
TS (Figure 7E). Wiener filter predictions on the test data (50% 
of trials) was computed and averaged to generate the mean pre- 
dicted trajectory for each TS (Figure 7F). Decoded neural activity 
reproduced the average trajectory with high accuracy (monkey 
N: SNR > 18 dB; monkey M: SNR > 10.3 dB) and the predictions 
even matched subtle behavioral effects such as the elongation 
of RT. 

We next asked if neural predictions of TS, RT, and movement 
kinematics depended on the time in the task interval. This analysis 
elucidated instances when cortical activity represented different 
parameters. Predictions of RT, TS, and movement velocity were 
obtained using LDA (Figure 8). A 100-ms sliding window was 
incremented at 25 ms intervals along the time axis relative to either 
target onset (Figures 8A—C) or movement onset (Figures 8D-F). 
At each time point, the neural activity in the single 100 ms bin 
trained the classifier on 60% of trials in the beginning, middle, 
or end of a session to make predictions of the three parameters 
in the remaining 40%. Significance was reached if the predicted 
parameter lay beyond the 95% CI (one proportion z-test) cen- 
tered on chance level performance. Aligned on target onset, there 
was significant encoding of both RT and TS with peak occurring 
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at 325 ms after target onset led to a significant local maximum but 
the representation of TS was much less prominent than in M1. 
Approach velocity was also found to be transiently represented 
in sensorimotor cortex of both monkeys. The temporal occurrence 
of the neural representation was much later, 650 ms after move- 
ment onset in monkey N M1 and 700 ms after movement onset in 
monkey M M1. Monkey M S1 representation was slightly different, 
with two distinct peaks, one occurring at 125 ms and the other at 
725 ms after movement onset. TS was not consistently found to be 
encoded after movement onset. As the data in Figures 8D-F are 


FIGURE 7 | Movement kinematics can be decoded from neural activity. 
(A,B) Movements along the X axis and Y axis decoded offline and shown 
with the actual traces. (C,D) Average X’ vs. time profile for the two velocity 
groups, both actual (solid lines) and predicted (dashed lines). (E,F) Actual X’ 
position vs. time for each target size, in both monkeys (E) compared with 
predicted X’ trace for each target size (F). Shown separately for clarity, 
however SNR computed by comparing actual and predicted for a given 
target size. In all predicted X’ trajectories, the single trial kinematics were 
predicted then averaged across the session to generate the traces in 
panels (C-F). 
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FIGURE 8 | Velocity, target size, and reaction time predicted using 
linear discriminant analysis. Each parameter was divided into three 
groups for analysis. Data for each prediction collected from a single bin, 
100 ms sliding window of neuronal data incremented at 25 ms through 
the specified interval. Data denote normalized fraction correct by dividing 
the fraction correct prediction by the chance level performance (see 
Materials and Methods). (A-C) Prediction of the three parameters 
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aligned on target onset (dashed vertical line) for monkey N M1 (A), 
monkey M M1 (B), and monkey M S11 (C). Noted on each is the mean 
time of movement onset (jo) with the mean + SD denoted by smaller 
black vertical bars on time axis. (D-F) Prediction of the three parameters 
now aligned on movement onset (dashed vertical line). Each panel 
shown with 95% confidence interval for expected LDA classification 
performance (gray horizontal band). 


aligned on movement rather than target onset, the pre-movement 
encoding of TS (p < 0.05) is likely a distorted representation of 
the results in Figures 8A—C. Taken together, there was a clear 
neural representation in sensorimotor cortex of RT and TS fol- 
lowing target onset. Separate to this representation is the velocity 
tuning of neurons in M1 and S1 occurring well after movement 
onset. 


DISCUSSION 

We elucidated the neural correlates underlying the changes in the 
parameters of arm reaching movements with TS. At the behav- 
ioral level, we found a strong TS-dependency (ID-dependency) 
for three parameters of arm reaches performed by rhesus mon- 
keys: total MT, RT, and mean approach velocity. At the neuronal 
level, analysis across the recorded neuronal populations in M1 and 
S1 revealed changes in modulations that depended on TS. These 
neuronal modulations could not be only explained by changes in 
movement parameters — the effect that would be expected if M1 
and S1 represented only motor aspects of that behavior. Rather, 
sensorimotor cortical activity represented both characteristics of 
movement and TS. During the RT period, the slopes of neuronal 
rates depended on both TS and RTI, and during movements neu- 
ronal rates depended on TS and movement velocity. Likewise, RT, 
TS, and cursor trajectories were each effectively decoded from cor- 
tical activity using both a Wiener filter and LDA predictions. These 
results indicate that M1 and S1 ensembles encode TS during motor 
preparation and execution. 


FITTS’ LAW IN MONKEYS 

There are clear similarities between the behavioral data obtained 
in the present study and those obtained from previous work in 
humans. We found a positive, linear relationship between ID and 
MT, confirming the canonical Fitts’ law relation (Fitts, 1954; Card 
et al., 1978). From regression of this data, the IP was computed for 
the two monkeys to be 8.21 b/s (monkey N) and 21.35 b/s (mon- 
key M). These compare to similar values shown for human motor 
systems, which Fitts found to be around 10-12b/s (Fitts, 1954). 
Monkey behavior also matched previous human work with respect 
to target-dependent shifts in RT (Munro et al., 2007; Boyd et al., 
2009) and velocity (Plamondon and Alimi, 1997; Munro et al., 
2007). 

When comparing monkey data to previous human findings, 
several key differences were discovered. We found that monkeys 
treat different movement directions unequally. For each monkey, 
one out of four directions was the most difficult and resulted in 
longer RTs and clearer Fitts’ law dependency compared to other 
reach directions. For other directions the monkeys moved in a bal- 
listic fashion, showing less clear features of Fitts’ law. We propose 
that ballistic movements can at least partially explain such loss of 
Fitts’ law-type behavior, in agreement with previous work on Fitts’ 
law in humans (Juras et al., 2009). 


NEURAL CORRELATES OF FITTS’ LAW 
Our data suggest that the neural correlates of Fitts’ law are 
observed in M1 and S1 during both motor preparation (RT 
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period) and movement execution (movement period). During 
motor preparation, populations of M1 and S1 exhibited clear mod- 
ulations of FR that were characterized as FR slopes. The slopes 
increased with shorter RTs and larger TSs. FR slopes have been 
shown to be involved prominently in decision making, as demon- 
strated by evidence-accumulation models (Roitman and Shadlen, 
2002; Gold and Shadlen, 2007). These models are based mostly 
on the data from eye-movement studies that required selecting 
a saccade direction in response to complex visual clues (Pearson 
and Platt, 2009; Resulaj et al., 2009). Thus, it has been shown 
that neural activity recorded in middle temporal (MT) and lateral 
intraparietal (LIP) areas rises more gradually when the task is per- 
ceived to be more difficult, thus lengthening the pre-movement 
period (Roitman and Shadlen, 2002; Churchland et al., 2008). 
Such dependency is similar to the behavioral and neurophysio- 
logical results of our study: we also observed elongated RTs for 
smaller TS, and the changes of neuronal rates developed slower. 
Our present findings also correspond to our previous study of 
self-timed movements where we observed lower FR slopes in M1 
and dorsal premotor (PMd) neurons for longer self-timed inter- 
vals (Lebedev et al., 2008). Thus, gradual changes of FR during 
behavioral epochs preceding movements appear to be a general 
phenomenon in the cortex during tasks that involve sensorimotor 
transformation and decision making. Notably, we did not observe 
a clear segregation between motor (M1) and sensory (S1) areas 
which both showed modulations during the RT period. This adds 
to previous reports of premotor activity in $1 (Soso and Fetz, 
1980; Lebedev et al., 1994). The exact role of M1 and S1 circuitry 
in the sensorimotor transformation that involves TS processing is 
not clear. One possibility is that M1 and S1 modulations reflect 
the processing performed mostly by associative areas (‘Thaler and 
Goodale, 2011) and basal ganglia (Lee and Assad, 2003). Alterna- 
tively, M1 and S1 may be involved in the sensorimotor transforma- 
tion as an essential part of a distributed network where there is no 
clear-cut segregation of processing modules (Shen and Alexander, 
1997; Zhang et al., 1997; Hernandez et al., 2010). While our results 
cannot distinguish between these two possibilities, the presence 
of TS information in M1 and S1 indicates that these areas are 
not involved in merely movement production, but reflect sensory 
components of the visuo-motor transformation, as well. 

The second feature of movements clearly reflected by corti- 
cal activity was velocity. Approach velocity was found to shift as 
a function of TS (Figures 3C,D). Neural activity at the time of 
movement initiation paralleled this shift via FR amplitude. Higher 
velocity trials correlated with higher FR during and slightly after 
movement onset (Figure 5D). This result was expected, as numer- 
ous previous studies have strongly linked motor cortical activity 
with velocity (Moran and Schwartz, 1999; Todorov, 2000; Lebe- 
dev et al., 2005; Wang et al., 2007; Li et al., 2009). Additionally, 
a large percentage of M1 neurons were found to encode both 
target-centric (reach distance) and motor (direction, position) 
parameters during movement, often in a serial manner (Fu et al., 
1995). Our findings also indicate that $1 neurons exhibit veloc- 
ity tuning, consistent with previous studies that have described 
S1 neurons with kinematic modulations (Gardner and Costanzo, 
1981; Cohen et al., 1994; Lebedev et al., 1994, 2005; Carmena 
et al., 2003). In our experiments, S1 representation of velocity was 


somewhat weaker than in M1, however the maximum modulation 
epoch was similar (300-700 ms after movement onset). 

Target size has not previous been identified to modulate FR 
profiles in M1 or S1. Using both linear regression and LDA, we 
found that in both M1 and S1 the neural representation of TS 
becomes prominent first during the RT period, coinciding with 
the rise of RT encoding (Figures 8A-C). This, again, expands the 
role of M1 beyond simply motor execution (Shen and Alexander, 
1997; Zhang et al., 1997; Merchant et al., 2004). Our data showed 
a second, more subtle effect in both monkeys, with TS again rep- 
resented 100-300 ms following movement onset (Figures 8D-F) 
near the onset of velocity representation. Thus, our results suggest 
that M1 is one of the loci of evidence-accumulation, since it seems 
to integrate TS information with motor parameters during the 
pre-movement period (affecting RT) and near movement onset 
(affecting velocity). 

Somewhat surprisingly, we did not observe clear-cut differences 
between M1 and S1 modulations during the execution of center- 
out reaching movements. During the pre-movement period, we 
found S1 neurons to modulate very similarly to M1. RT and TS 
were both encoded in S1 during this period, although the onset of 
this representation was no sooner than 200 ms after target onset, 
compared to an earlier 50-100 ms post-stimulus representation in 
M1 (Figures 8B—C). Pre-movement activity has been previously 
reported in S1 (Soso and Fetz, 1980; Nelson et al., 1991; Lebedev 
et al., 1994). Moreover, visual information has been reported to 
affect pre-movement activity patterns in S1 (Liu et al., 2005). Our 
finding of TS representation in M1 provides additional evidence 
of the representation of visual information in M1. In addition to 
cortical visual streams (Wise et al., 1997), the basal ganglia appear 
a likely candidate for a structure that mediates transmission and 
processing of this information (Alexander and Crutcher, 1990; Lee 
and Assad, 2003; Opris et al., 2011). 


IMPLICATIONS FOR BMIs 
Much current BMI research focuses on improving movement pre- 
dictions by either technical improvement (electrode type/number) 
or algorithm optimization. The motor goal for cortically con- 
trolled neuroprosthetics is to recreate complex naturalistic move- 
ments using only the neural activity for the patient. Notwith- 
standing strong early work toward this goal (Carmena et al., 2003; 
Lebedev et al., 2005; Moritz et al., 2008; Velliste et al., 2008; Vargas- 
Irwin et al., 2010), there are still many milestones to accomplish 
(Lebedev and Nicolelis, 2006; Nicolelis and Lebedev, 2009; Lebedev 
et al., 2011). A recent human BMI study using EEG (P300) dri- 
ven BMIs confirmed Fitts’ law in movements controlled by neural 
signals, thus supporting the idea that Fitts’ law parameters can be 
incorporated into BMI cursor control (Felton et al., 2009). Because 
Fitts’ law is widely seen in naturally enacted movements, Felton and 
others (Gilja et al., 2011; Simeral et al., 2011) suggest that Fitts’ law 
would be an effective tool for comparing BMI subjects, modali- 
ties, and tasks. With the results from the present study, we suggest 
that TS, in addition to motor parameters could be decoded from 
neural activity in order to improve neuroprosthetic control and 
approximate naturalistic movements. 

The present understanding of Fitts’ law has drastically influ- 
enced the ergonomics field, especially in the streamlining of 
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human-computer interaction. Computer interface designs heav- 
ily favor rapid point-and-select layouts, prompting the prevalence 
of pop-up and pie menus, corner icon locations, and appropriate 
sizing of buttons and GUI controls (MacKenzie, 1992; MacKenzie 
et al., 2001). Pointing devices such as a mouse or joystick were 
demonstrated to follow Fitts’ law in a similar manner as manual 
pointing (Card et al., 1978). The International Organization of 
Standards requires compliance with Fitts’ law optimized perfor- 
mance on all non-keyboard computer input devices, quantified by 
both IP (in bits per second) and error rate (ISO9241-9:2000(E), 
2002). As brain—computer interfacing develops in coming years, 
it will be increasingly important to understand the underlying 
neural mechanisms behind this behavioral property. With BMIs, 
the subject’s thoughts replace the mouse or joystick as the input 
device. Being able to decode the size of the desired target, for exam- 
ple, enables a forward model to enhance performance in terms of 
accurate on-line kinematic predictions, indices of performance, 
and decreased error rates — the ISO criteria currently used for 
input devices. Such a forward model could extract the TS from 
the neural activity during the RT period, as was demonstrated in 
the present study, and use this parameter to guide more accurate 
brain-controlled movements. This result would greatly impact the 
field of neuroprosthetics and make headway toward realizing the 


The results from this study demonstrate the existence of Fitts’ 
law in two rhesus macaque monkeys and provide strong indi- 
cations of the underlying neural correlates. The changes at the 
behavioral level were paralleled by the modulations of M1 and 
S1 neurons during the pre- and PME. TS-dependent modulation 
existed in addition to kinematic tuning thus suggesting a potential 
encoding that could be exploited in the design of future BMIs. 
Improved movement predictors that incorporate reach informa- 
tion such as TS will enhance the ability of cortically driven neuro- 
prosthetics in terms of both accuracy and similarity to naturalistic 
movements. 
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INTRODUCTION 


The ability to inhibit unwanted movements and change motor plans is essential for 
behaviors of advanced organisms. The neural mechanisms by which the primate motor 
system rejects undesired actions have received much attention during the last decade, 
but it is not well understood how this neural function could be utilized to improve 
the efficiency of brain-machine interfaces (BMls). Here we employed linear discriminant 
analysis (LDA) and a Wiener filter to extract motor plan transitions from the activity of 
ensembles of sensorimotor cortex neurons. Two rhesus monkeys, chronically implanted 
with multielectrode arrays in primary motor (M1) and primary sensory (S1) cortices, were 
overtrained to produce reaching movements with a joystick toward visual targets upon 
their presentation. Then, the behavioral task was modified to include a distracting target 
that flashed for 50, 150, or 250ms (25% of trials each) followed by the true target that 
appeared at a different screen location. In the remaining 25% of trials, the initial target 
stayed on the screen and was the target to be approached. M1 and S1 neuronal activity 
represented both the true and distracting targets, even for the shortest duration of the 
distracting event. This dual representation persisted both when the monkey initiated 
movements toward the distracting target and then made corrections and when they 
moved directly toward the second, true target. The Wiener filter effectively decoded 
the location of the true target, whereas the LDA classifier extracted the location of 
both targets from ensembles of 50-250 neurons. Based on these results, we suggest 
developing real-time BMls that inhibit unwanted movements represented by brain activity 
while enacting the desired motor outcome concomitantly. 


Keywords: motor cortex, sensorimotor transformation, volitional inhibition, neurophysiology, decision making, 
brain-machine interfaces, neuroprosthetics, monkey 


and reprogram movements once the corresponding neural plan- 


Neurophysiological studies conducted during the last two decades 
have revealed a complex representation of spatial information 
in the brain, including the representation of multiple motor 
targets (Cisek and Kalaska, 2002, 2005), sequences (Mushiake 
et al., 1990; Isoda and Tanji, 2004), spatial attention (Lebedev 
and Wise, 2001; Lebedev et al., 2004; Ikkai and Curtis, 2011), 
and gaze (Boussaoud et al., 1993; Baker et al., 1999; Boussaoud 
and Bremmer, 1999; Balan and Ferrera, 2003)—all in differ- 
ent reference frames, depending from which brain area neural 
activity was sampled (Lacquaniti and Caminiti, 1998; Cohen and 
Andersen, 2002; McGuire and Sabes, 2009). These representations 
underlie rich behavioral repertoires of advanced organisms, pri- 
mates in particular, that can flexibly control their attention and 
motor processing to meet demanding challenges of their envi- 
ronments (Wise et al., 1996; Wise and Murray, 2000; Lebedev 
and Wise, 2002). In particular, advanced organisms can inhibit 


ning or even the movement itself have been initiated (Matsuzaka 
and Tanji, 1996; Band and van Boxtel, 1999; Schall et al., 2002; 
Mostofsky and Simmonds, 2008; Verbruggen and Logan, 2008; 
Stinear et al., 2009; Mirabella et al., 2011). 

An adaptive neural framework can enable the planning stages 
of potential movements to begin in parallel with preparations for 
an alternative motor plan (Resulaj et al., 2009; Cisek and Kalaska, 
2010). As a result, neural representations of distinct motor plans 
may compete prior to movement onset in behavioral tasks with 
several potential targets of movement (Cisek and Kalaska, 2005; 
Rickert et al., 2009; Mirabella et al., 2011). Studies of reaching 
movements have identified populations of neurons that represent 
multiple potential motor plans throughout the dorsal premotor 
(Cisek and Kalaska, 2005; Pesaran et al., 2008; Mirabella et al., 
2011), supplementary motor (Chen et al., 2010), and posterior 
parietal cortices (Snyder et al., 1998; Scherberger and Andersen, 
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2007). A bounded-accumulation model (Resulaj et al., 2009) pro- 
poses that when multiple motor outcomes are presented, neural 
networks prepare for the most likely upcoming movement. The 
network accumulates noisy evidence over time until a bound 
threshold is reached, at which point an initial decision is reached, 
which is then either reversed or reaffirmed. 

In the present study, we investigated the representation of 
multiple potential movement targets and the specification of a 
change in motor plan by neuronal ensembles simultaneously 
recorded in primary motor (M1) and sensory (S1) cortical areas. 
We approached neural representation of motor plan transitions 
from a brain-machine interface (BMI) perspective. BMIs extract 
motor commands from the brain and convert them into move- 
ments of external actuators, such as computer cursors and robotic 
devices (Andersen et al., 2004; Lebedev and Nicolelis, 2006; Fetz, 
2007; Birbaumer et al., 2008; Nicolelis and Lebedev, 2009; Chase 
and Schwartz, 2011; Lebedev and Nicolelis, 2011; Lebedev et al., 
2011). At the current stage of the BMI field, up to several hun- 
dred neurons in the brain can be recorded simultaneously by 
chronically implanted multielectrode arrays (Nicolelis et al., 2003; 
Chapin, 2004; Churchland et al., 2007; Miller and Wilson, 2008; 
Lebedev and Nicolelis, 2011; Lebedev et al., 2011; Stevenson and 
Kording, 2011). Recording from large neuronal populations is 
essential because the range of information extracted from neu- 
ral activity and accuracy of extraction improves with the number 
of recorded neurons (Wessberg et al., 2000; Carmena et al., 2003; 
Lebedev et al., 2005; Lebedev and Nicolelis, 2006; Fitzsimmons 
et al., 2009; Nicolelis and Lebedev, 2009). Notwithstanding the 
successes of the BMI field, signals extracted from the brain are 
typically noisy (Lebedev and Nicolelis, 2006; Tonet et al., 2008). 
BMI algorithms are usually trained to reproduce one particu- 
lar behavior and do not generalize well when a transition to 
a new set of rules and conditions is needed (Santucci et al., 
2005; Fitzsimmons et al., 2009). This is why many improve- 
ments are needed: from a significant increase of the number of 
simultaneously recorded neurons to the development of better 
extraction algorithms capable of approximating natural behav- 
iors. 

Although some work has been done on the extraction of 
behavioral parameters during delay intervals, during which mon- 
keys prepare movements but withhold their execution (Musallam 
et al., 2004; Lebedev et al., 2008; Afshar et al., 2011), the prob- 
lem of motor plan transitions has not yet been fully investigated 
from a BMI perspective. We examined cortical representation of 
motor programming in a reaction-time task in which monkeys 
had to rapidly reprogram their center-out reaching movements. 
The monkeys had been previously overtrained to move a hand- 
held joystick toward computer screen targets. In this study, we 
introduced distracting targets that flashed on the screen for a 
short period (50-250 ms) and triggered motor preparation on 
75% of the trials. This motor preparation had to be canceled when 
a true target appeared at a new screen location. Both the distract- 
ing and the true targets were represented by neuronal ensemble 
activity recorded in the M1 and S1 cortices. We used ensemble 
modulations to extract target locations using a linear discrimi- 
nant analysis (LDA) classifier. In addition, a Wiener filter was used 
to make continuous extractions offline. 


METHODS 

CORTICAL IMPLANTS 

All studies were conducted with approved protocols from the 
Duke University Institutional Animal Care and Use Committee 
and were in accordance with the NIH guidelines for the Care 
and Use of Laboratory Animals (National Research Council et al., 
2011). 

Two rhesus monkeys (one male and one female, monkeys M 
and N, respectively) were chronically implanted with multielec- 
trode arrays in M1 and S1 of both right and left hemispheres using 
previously described surgical methods (Nicolelis et al., 2003). 
Within each hemisphere, two 96 channel microelectrode arrays 
were placed in cortical areas corresponding to cortical repre- 
sentations of the arm and leg (Figure 1B), but in this study, 
neural activity was recorded only in the arm representation area 
of right hemisphere M1 (in both monkeys) and S1 (only in 
monkey M). Each array consisted of two 4 by 4 grids of inde- 
pendently movable electrode triplets. Triplets were comprised 
of electrodes of different lengths, in 0.3mm intervals, which 
allowed us to sample neuronal activity from different depths in 
the cortical tissue. Recorded signals were amplified, digitized, and 
filtered by a multichannel recording system (Plexon Inc, Dallas, 
TX, USA). Neuronal spikes were sorted using waveform tem- 
plate matching algorithm built into the real-time spike-sorting 
software. 


BEHAVIORAL TASK 

Each monkey was trained to move a hand-held joystick to control 
the two-dimensional location of a computer cursor on a screen 
(Figure 1A). X(left-right) and Y(forward-backward) position of 
the joystick were translated to X(left-right) and Y(up-down) cur- 
sor position on the screen. The joystick was affixed to the chair at 
the waist level of the monkey on the side of the working hand (left 
hand in both monkeys). During the task, the display screen was 
positioned in front of the monkey, at 45cm from the monkey’s 
eyes. The cursor diameter was 0.5 cm. 

To begin each trial, the monkey placed its left hand on the 
top of the joystick, causing a cursor to appear on the screen. 
A trial was immediately canceled if the monkey removed its 
hand from the joystick at any time. Next, a 5cm diameter circle 
appeared at the center of the screen. The monkey moved the cur- 
sor inside this circle and held it for a random interval between 
1 and 2s. After this hold period, the center target disappeared 
and a single peripheral target became visible. The peripheral 
target appeared as a thickened 40° arc on a thin boundary cir- 
cle aligned on the center of the screen (Figure 1C). Reaching 
the target required the cursor to pass over the thickened arc 
from the inside of the circle, moving outwards (Figure 1D). If 
the cursor crossed the circle, but missed the target, trial was 
terminated without reward delivery. Both monkeys had been pre- 
viously overtrained to perform center-out movements toward 
the targets, triggered by target appearance and characterized by 
reaction times (RTs) of 0.49 + 0.17s (mean + standard devia- 
tion) in monkey M and 0.44 + 0.18s in monkey N (ifft et al., 
2011). 

In this study we introduced target switches to produce dual tar- 
get representation in the sensorimotor cortex. This design was to 
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FIGURE 1 | Experiment and location of neural recordings. (A) Rhesus 
monkey controlled joystick with left hand which translated to movements 
on computer screen. (B) Electrode arrays in arm representation regions of 
M1 and S1 cortex were implanted prior to data collection. (C) During task, 
peripheral targets appeared at one of four potential locations on the screen: 
0, 90, 180, 270°. Workspace was divided into four quadrants, centered on 
each target location as divided by dashed diagonal lines. (D) Typical task 
sequence begins with cursor inside central target. After a random hold 
time, the target appears an as an arc on the gray boundary circle. On 25% 
of trials, this target persists and the cursor must be move through the 
target for reward. On the remaining trials, the target moves after a brief 
delay and the cursor must be move toward the new target to obtain a 
reward. (E) Shown are timelines of the presence of each target and joystick 
position. SSD is defined to be the time between when the first target 
appears and when the target is switched to the second location. 


mimic the momentary preparations followed by changes in motor 
plans. In switch trials, initial targets served as distractors. They 
appeared on the screen and were then replaced by a second target 
at different locations after a short interval, termed the switch sig- 
nal delay (SSD). A similar switching task was previously reported 
(Georgopoulos et al., 1981, 1983) with a difference that monkeys 
were overtrained in those studies and M1 neurons were recorded 
serially. Our distracting and true targets always appeared at one 


of four locations on the screen, at angles 0, 90, 180, or 270° rela- 
tive to the center of the screen (Figure 1C). Initial targets switched 
in 75% of trials. The SSD for a given trial was either 50, 150, or 
250 ms, with each occurring with equal probability. When the dis- 
tracting target disappeared, a second target appeared at one of 
the remaining three potential locations. We call this second tar- 
get the true target because a juice reward was obtained only by 
moving the cursor through this target. In the remaining 25% of 
trials, the first target remained on the screen throughout the trial 
and a reward was obtained by passing the cursor through this 
target location. Once the true target appeared, the monkey had 
2.5 s to complete each trial before timing out. The experiment was 
repeated over two daily recording sessions in both monkey M and 
monkey N. 

Single trial trajectories were categorized depending on the 
degree of deviation made toward the distracting target. For switch 
trials, a threshold distance was set at 1.5 cm along the axis between 
the center target and the distractor. Joystick movements which 
surpassed this threshold in the direction of the distractor were 
categorized as distracted trials. Remaining trials were categorized 
as direct if, in addition to not moving toward the distractor, the 
path to the true target deviated less than 1.5 cm in the direction 
orthogonal to the ideal trajectory. Strict criteria were enforced for 
direct trials to ensure that the only movement made was to the 
true target, isolating the role of movements with a singular goal 
from the onset. Direct trials could, however, be unrewarded if 
they were near misses and the cursor did not move toward the 
distractor. Violation of these criteria on a given trial resulted into 
classification as a distracted trial and later analyses evaluated these 
two trial groupings separately. Furthermore, trials where the ini- 
tial center target was held but no movement was made to any 
target (beyond the 1.5cm threshold distance) were not consid- 
ered in the present analysis. Trial movement onset was computed 
using a previously implemented algorithm where movement ini- 
tiation was detected based on the analysis of specific patterns in 
velocity and acceleration (Ifft et al., 2011). To perform statistical 
testing on performance accuracy measured as proportion of tri- 
als, such as fraction direct trials (Figure 2C) or fraction correct 
for different target location, trials were subdivided into groups of 
15 trials. Number of outcomes of each type per 15 trials was com- 
puted for each group, and their statistical sample for all groups 
was entered in an appropriate statistical test (e.g., unpaired t-test 
that could be used either directly to compare two outcomes, 
or post-hoc following analysis of variance for comparing several 
outcomes). 

RT was defined as the time from either the distractor or true 
target onset until movement onset. The RT measured from dis- 
tractor onset was elongated by the SSD during which an initial, 
false, target was presented. At the same time, it could be addi- 
tionally shortened for some SSDs because the distractor primed 
the appearance of the true target. That is, when the true tar- 
get was presented, a shorter response latency could indicate that 
the transient distractor presence expedited pre-movement pro- 
cesses with respect to the true target. For the RT measured from 
true target onset (Figure 1E), the priming effect of the distrac- 
tor is more clearly demonstrated. Each definition yields its own 
interpretation thus we included both in our analysis. 
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POPULATION RESPONSES 

All neural analyses were performed using recordings from 
Monkey M M1 and S1 neurons, and Monkey N M1 neurons. 
Population-level analyses were conducted separately for each of 
these three cortical areas. Neural activity was analyzed using 
peri-event time histograms (PETHs) (Awiszus, 1997) aligned on 
distractor target onset. Recorded timestamps of action poten- 
tials were counted in bins of 50 ms width. The PETH from 0.5 s 
before to 1.0s after distractor target onset (true target onset 
in the case of no-switch trials) for each neuron was computed 
separately for each combination of distractor and true target loca- 
tion (four no-switch combinations and 12 switch combinations), 
for each of the four SSD conditions. For single neuron analy- 
sis (Figures 4, 5), single trial spike rasters were constructed over 
the same 1.5s epoch aligned on distractor onset. Corresponding 
PETHs represent the bin counts of total spikes that occurred 
within each of the 50 ms bins, summed across trials in the same 
SSD category as well as matching the same combination of dis- 
tractor and true target locations. Bin counts were then divided 
by the fixed 50 ms bin width to represent firing rate in units of 
spikes/second. 

To analyze population-level modulations, the average modu- 
lation profile for each neuron was normalized by subtracting the 
mean bin count of the neuron (over all conditions) and dividing 
by the standard deviation of the neuron’s bin count. This normal- 
ized quantity represented modulations as a fraction of the overall 
modulations, or statistically, the z-score. The directional tuning of 
each neuron was computed from normalized PETH data on trials 
where there was no switching of the peripheral target. The mean 
normalized firing rate was computed within the 750 ms window 
following target onset for each target direction. The four direc- 
tions were then ranked from most preferred to least preferred in 
subsequent analyses reflecting the directional preference of the 
each neuron (Figure 6A). Next the mean PETH over the entire 
population of neurons was computed for each of the sixteen dis- 
tractor/true target configurations and for each of the three SSD 
groups (Figure 6B). 

To further understand the neural representation of the dis- 
tractor target and the true target, the population mean firing 
rate (MFR) was computed during different epochs for each of 
the 12 distractor-true target combinations, and four no-switch 
trial categories (Figure 7). Furthermore, we separated the trials 
by SSD to elucidate the effect of an elongated distractor presence 
(Figures 7A—C). For each neuron, the MFR was computed for 
each of the 16 positions on a 4 x 4 grid, with rows representing 
preference ranking of the true target location (ranks 1 through 4) 
and columns representing the preference ranking of the distrac- 
tor target (ranks 1 through 4). The layout is clarified in Figure 7D. 
Firing rate was normalized and the directional preferences of each 
neuron were determined from most to least preferred direction 
as in Figure 6. Population MFR was obtained by averaging the 
MER across all neurons in the given area. This procedure was 
repeated for six temporal epochs: the epoch when the distrac- 
tor was present, and five consecutive 100 ms epochs following 
true target appearance. To evaluate the specific contribution of 
the distractor and true target neural representations on MFR, we 
fit MFR as a linear function of the preference ranking of each 


target preference combination (1-4) for each 4 x 4 grid, show in 
Equation 1 (Lebedev and Wise, 2001; Lebedev et al., 2004): 


MER = A: PDtrue + B+ PDaistractor + C (1) 


Coefficients A and B represent the contribution of the true target 
and the distractor target, respectively. 

We also separated trials depending on whether the monkey 
correctly switched to the true target (Figure 8). In this analysis, 
trials were separated into two groups depending on the monkey 
behavior: (1) trials where the true target was reached (rewarded 
trials) and (2) trials where the monkey was distracted (as defined 
earlier) and failed to reach the true target. Trials outside of these 
two categories were not included in the Figure 8 analysis. 

To evaluate the variation in neural activity profile between the 
different SSD groups, single neuron normalized PETH data from 
each of the 12 switch conditions were subtracted from the PETH 
data from the corresponding no-switch condition and this value 
was squared. The mean of the 12 difference-squared terms was 
computed over the —0.5 to 1s trial epoch for each neuron and 
for each SSD group, and the square root of this value was com- 
puted, yielding a root-mean-square (RMS) difference, shown by 
Equation 2: 


RMS,e = |mean { )* (PETH, — PETH,)” (2) 
idj 


where PETH;; represents the normalized neural activity profile 
for a single neuron when the distractor is at position i and the true 
target is at position j. PETH; represents the normalized activity 
profile of the same neuron on a no-switch trial. Both 7 and j have 
four possible values resulting in 12 differences to be computed for 
each neuron. This procedure was repeated for data collected in 
each of the four SSD groups. 

The difference profile across the population was thus com- 
puted by taking the mean difference across neurons, while main- 
taining temporal information (Figure 9A). Lastly, the population 
average for each of the SSD groups was compared to identify the 
relevant interval during the trial where modulations reflect the 
transient distractor representation (Figure 9B). 

To assure that the differences arose as a result of the distrac- 
tor, and not due to increased variance during elevated neural 
activity during movement, trials were shuffled amongst distractor 
location groups and the analysis was repeated, however, the SSD 
categorization remained intact. Once shuffled, the single neu- 
ron and population RMS differences were computed in the exact 
method as performed for the unshuffled data. The shuffled RMS 
difference profile for each SSD was generated five times and the 
average of these profiles was subtracted from the unshuffled pop- 
ulation RMS difference profile, thus reflecting the true difference 
accounted for by the distractor presence. 


CLASSIFIER 
To extract the location of both the first (distractor) target loca- 
tion and the second (true) target locations, LDA (Fisher, 1936) 
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was used to decode neural activity offline and make categorical 
predictions of each target location (Figures 9-11) (Ifft et al. 
2011). In no-switch trials, the locations of the distractor and 
true targets were considered the same. Neural activity aligned 
on distractor onset was used to train the classifier on 60% of 
randomly selected trials. Training data for each time point was 
provided by neural discharges within a 150 ms window slid across 
the task interval from 0.5s before to 1s after distractor target 
onset. Predictions of both target locations were made using sam- 
ple data from the remaining 40% of trials. The same 150 ms 
sliding window was used to obtain predictions. For each ses- 
sion, sliding LDA predictions were computed five times, each with 
randomly redrawn training and sample subsets. Overall reported 
predictions represent the average of these five runs per session. 
Neural activity used to train the decoder was separated into exclu- 
sively M1 or S1 recorded neurons in the case of monkey M, and 
just M1 neurons in monkey N. LDA predictions were also made 
with shuffled data; that is, when the group information is ran- 
domly permutated prior to training the LDA classifier. For each 
LDA figure (Figures 10-12), we computed the fraction correct 
prediction of each parameter minus fraction correct of the LDA 
predictions from shuffled data. The chance level performance 
(0.25 because of four potential targets) was then added to this 
amount to again return to the conventional [0, 1] scale. The y- 
axis thus becomes fraction correct with unrelated modulations 
removed. For each analysis, confidence intervals were computed 
using the 1-proportion z-test from Equation 3: 


P—Po 
= 3 
° VPpol =a 7 


where po is 0.25 (four potential targets) and n is the number of 
trials used for testing (40% of total trials). All confidence intervals 
were constructed at a = 0.05. 

Trial types were then divided according whether the movement 
was direct to the true target or revealed a deviation toward the 
distractor (see Behavioral Task section above) as a way to test 
whether the transient representation of the distractor target is 
explained by motor movements. LDA was trained on both trial 
types combined and was then utilized to make predictions of 
the first and second target location separately for each trial type 
(Figure 11). 

A separate analysis was performed using LDA to decode the 
presence of the switching of target location (Figure 12A). Again, 
a 150 ms sliding window of neural activity trained the decoder. At 
each time step, LDA made a prediction of whether the trial was a 
switch or no-switch trial. Data were again aligned on the time of 
distractor onset and all switch-trials were grouped together. First 
target in this case means distractor target in the case of switch 
trials, or true target (only target) in the case of no-switch trials. 
The 150 ms sliding window was incremented along the time axis 
in 25 ms steps from 0.5s before to 1s after first target onset. The 
fraction of correct predictions was computed at each time step as 
described for previous sliding window LDA analyses. As this anal- 
ysis involves a binary classification procedure, a second metric 
was used to quantify extraction of event information from neu- 
ral activity (Figure 12B). The Matthews correlation coefficient 


(MCC) is a common measure of classifiers for binary outcomes 
(Matthews, 1975; Baldi et al., 2000). MCC is computed as shown 
in Equation 4: 


TP x TN — FP x FN 
MCC = (4) 
(TP + EN)(TP + FP)(TN + FP)(TN + EN) 


where TP is the count for true positives predictions, TN for true 
negatives, FP for false positives, and FN for false negatives. Similar 
to the conventional correlation coefficient, the values of MCC 
range from —1 to 1 depending on the strength and directionality 
of the prediction. At each shift, the MCC was computed five times 
as a result of randomly redrawing the training and sample data, 
as was done in each LDA analysis. With two sessions per monkey, 
MCC at each time step of the sliding window represents the mean 
of 10 values. 

Concurrent to prediction of trial type (switch or no-switch) 
at each time step, location of the first and second targets was 
also decoded. Although the training data included both switch 
and no-switch trials, LDA performance in terms of fraction cor- 
rect locations only included prediction data from switch trials. 
This was necessary because if the no-switch target was identified 
as the distractor, the LDA predictions may have been artificially 
improved due to the prolonged presence of that target on the 
screen. If it was identified as the true target, the prediction may 
have also been improved because of the absence of interference 
from a distractor. 


CONTINUOUS OFFLINE PREDICTIONS USING WIENER FILTER 

To mimic continuous, real-time BMI predictions, we used a sim- 
ple Wiener filter with six 100 ms taps of neural data to predict 
cursor X and Y coordinates, and true target X and Y coordinates 
at a 10 Hz output rate (Figure 13). For monkey M, both M1 and 
S1 neurons were used to improve predictions of these parameters 
(Figures 13A—E). To reject noisy neurons and reduce overfitting, 
we computed weights that reflected each neuron’s contribution 
toward kinematic predictions. We selected the 80% of all neurons 
which had the highest weights. 

The Wiener filter weights were fit using 60% of the session 
length and predictions were made using the remaining 40%. Due 
to variable durations of all targets in the session, we inflated the 
true target duration to 1000 ms to ensure targets could be repre- 
sented despite a low 10 Hz rate of prediction. To reduce the effect 
of noisy predictions, the predicted radius of movement, r, was 
computed at each time as shown in Equation 5: 


p= (Xe + ¥5 (5) 


where X, and Y, are the predicted X and Y position of the cursor 
at a given time (Figure 13B). A threshold for r was chosen at 5 cm 
(screen coordinates) such that when r surpassed this threshold, 
a reach had been predicted. Time of threshold crossing was thus 
recorded and predictions of target locations relative to this time 
were made. At each time from when r exceeded frthreshold predic- 
tions of cursor and true target position were made and compared 
to actual. To quantify performance in terms of fraction correct, 
the screen was divided into four (90°) quadrants surrounding 
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each target (see Figure 1C). For example, quadrant 1 would occur 
from —45° to 45° relative to the center of the screen. A prediction 
of cursor X and Y was correct if the Wiener prediction was in the 
correct quadrant as the actual cursor (X, Y). True target location 
predictions were evaluated in the same way. Predicted true target 
quadrant was compared with the target locations. The fraction 
correct was computed at each time step beginning at threshold 
crossing until 800 ms after threshold crossing, during all thresh- 
old crossings in the last 40% of each session. If during this 800 ms 
window, r became less than rhreshold> predictions were no longer 
made. If r was greater than rthreshold» but the true target was not 
on the screen (with all durations fixed at 1 s, as stated before), this 
was counted as incorrect. 


RESULTS 

BEHAVIOR 

While the monkeys had been previously overtrained in the sin- 
gle target task, no prior training was performed in the task with 
switching targets. We chose to avoid excessive training on the 
two target sequence because we wanted to obtain the maximum 
effect of the switching target and avoid the monkeys developing 
alternative behavioral strategies, such as timing their responses 
in a way such that the initially presented target is wholly ignored. 
We suppose that previous overtraining in reaction-time responses 
toward single targets helped to enhance the representation of 
the distractor target because that was the target toward which 
monkeys were accustomed to moving in a RT manner. However, 
neither the contribution of prior training nor the effect of con- 
tinued training with switching targets were examined in this 
study. 

As it would be expected, the introduction of switching targets 
resulted in erroneous responses on a portion of trials (Figure 2), 
more so for monkey N. We consider reaching movements that 
crossed into the true target to be correct trials. Incorrect trials 
consisted of movements that crossed distracting targets, move- 
ments that missed both targets, canceled and time-out trials 
(Figure 2B). The overall proportion of correct trials was higher 
in monkey M than in monkey N: 88% in monkey M and 
only 27% in monkey N (p < 0.001; Wilcoxon rank sum test) 
(Figures 2B,C). Monkey N compensated for this inaccuracy by 
making many more trials per recording session (1519 and 1072 in 
each of two sessions, respectively) than monkey M (594 and 418 
trials). This difference in accuracy is seen in the example trajec- 
tories for a particular target configuration (Figure 2A). Monkey 
M produced straight and carefully targeted reaching trajecto- 
ries, whereas monkey N’s trajectories were less accurately directed 
and often missed the target. Monkey M made errors only for 
the longest, 250 ms, SSD (Figure 2C). Another observation was 
that 90° switch trials were less frequently direct, compared to 
180° switch trials (78.4% vs. 90.3% of trials for monkey M and 
55.9% vs. 70.2% of trials for monkey N, p < 0.01, unpaired 
t-test). In other words, the distractor had a stronger effect on the 
cursor trajectory when it was closer to the true target. 

To separate accurate and inaccurate reaches toward the true 
target, trials were divided into two groups consisting of direct 
trials and distracted trials. Classification of trials is described in 
detail within Methods. In the separation of all monkey M’s trials, 


505 trials were classified as direct, 145 classified as distracted, 
compared against a total of 214 no-switch trials. Joystick trajec- 
tories for these types of trials are shown in Figure 3 for monkey 
M. Trials are grouped by target configuration: with 90° separa- 
tion between the initial and true targets (left panels), and with 
180° separation (right panels). Our data show reach trajectories 
that deviate toward the distractor on the infrequent distracted 
trials. Inaccurately performed trials for long presentations of dis- 
tracting targets can be also seen in the examples of Figure 2A. 
The trial-averaged traces reveal the largest deviation during the 
longest, 250 ms, SSD (Figure 3E). In both monkeys, the frac- 
tion of trials categorized as “direct” decreased with longer SSDs 
(Figure 2C, P < 0.001, Kruskal-Wallis test). More precisely, there 
was a decrease in fraction direct trials for 150 ms and 250 ms dura- 
tions for monkey N (p < 0.05; post-hoc unpaired t-test) and for 
250 ms duration for monkey M (p < 0.05) in comparison with 
no-switch trials. Previous studies of an overtrained switching task 
reported mostly distracted trials where monkeys initiated move- 
ments toward the distractor and then curved the trajectory toward 
the true target (Georgopoulos et al., 1981, 1983). 

The effect of SSD on RT was dependent on how RT was defined 
(see Methods). When defined from distractor target appearance 
until movement onset, longer SSDs caused a lengthening in RT 
in both monkeys (p < 0.001, One-Way Kruskal-Wallis test). This 
is somewhat expected because with longer SSDs, the longer the 
monkey must wait for the true target, thus inflating the RT. 
However, when RT was defined relative to true target appearance 
(Figure 2D), longer SSD caused shorter RTs (p < 0.001, One- 
Way Kruskal-Wallis test). Thus, the appearance of the distractor 
on the screen primed the response to the true target, even though 
the directional information that it provided was incorrect. Post- 
hoc analysis revealed significant differences from no-switch trial 
RT among both the 150 and 250 ms SSD groups for monkey N 
(p < 0.001; Wilcoxon rank sum test) and among all three SSD 
groups in monkey M (p < 0.001). Mean RTs for monkey N (0.52 
+ 0.24s; mean + sd) were significantly longer than the RTs for 
the single target task (0.44 + 0.188; p < 0.001; Wilcoxon rank 
sum test), however, monkey M performance was similar in both 
experiments (two-target sequence RT: 0.46 + 0.10; single target 
task 0.49 + 0.17s for monkey M). Overall, monkey N behavior 
was more erratic in the present experiment, as evidence by cur- 
sor trajectories (Figure 2A) and RT standard deviation more than 
twice that of monkey M. In the previously overtrained reaches to 
single targets both monkeys performed well (e.g., 84% and 74% 
correctly performed trials in monkeys M and N, respectively) (Ifft 
et al., 2011). Note that data from the previous study represents 
center-out movements to a target less than half the size of the 
target used in the present study. 


NEURONAL RESPONSES 
The initial distracting targets were represented by M1 and S1 
modulations even when they appeared for a brief 50 ms inter- 
val. This representation became more pronounced with longer 
presentations of distracting targets. 

Figure 4 shows a representative M1 neuron recorded in mon- 
key M that had a clear directional preference for the 90° and 
180° target location and was modulated in response to both the 
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FIGURE 2 | Behavioral results from both monkeys. (A) Typical movement 
traces from one combination of distractor and true target locations. The 
columns show different cursor trajectories on trials with different SSDs. The 
first row is data from monkey N and the second row from monkey M. (B) Pie 
chart shows the breakdown in trial outcomes by monkey. (C) The fraction of 
direct trials shown for each SSD group in both monkey M (left) and N (right). 
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true target appearance to movement onset. Same statistical procedure 

as (C). 


distractor and true targets. Data are arranged in a 4 by 4 matrix 
representation (Lebedev and Wise, 2001; Lebedev et al., 2004) 
where columns of panels correspond to the distractor target loca- 
tion and rows correspond to the true target location. The panels 
on the diagonal (shaded in gray) correspond to trials where the 
first target did not disappear and was the true target to which 
the monkey had to move. Modulations reflecting the distractor 
location can be appreciated from the comparison of the responses 


within the same rows of panels, but for different columns. 
Modulations reflecting the true target location are seen within 
the same columns, but for different rows. With the exception 
of the diagonals, which are identical in both Figures 4A and B, 
Figure 4A shows data for the 50 ms duration of the distractor, and 
Figure 4B shows data for the 150 ms duration. The responses to 
the distractor target are mostly clear in Figure 4B where bursts 
of activity are seen in response to that target appearing at the 
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FIGURE 3 | Raw cursor trajectories for two sessions (1018 total trials) 


no-switch trials. Offline, all targets were moved to (X,Y) position (5,0) and 
the associated coordinate transform was made to all kinematic data. Y 
indicates movement orthogonal to the ideal trajectory. (B) For switch trials, 
two categories of trials shown separately for clarity: trials with a 90° switch 
(left) and trials with a 180° switch (right). The coordinate systems for a 


direction and the Y direction was orthogonal to this axis. (C) X and Y cursor 
positions versus time for direct trials among 90° switch (left) trials and 180° 
switch trials (right). (D) Same as (C) except looking at only distracted trials. 
(E) Average X and Y trajectory of cursor separated by SSD (see Legend) 


preferred location (90°, less clear for 180°). Responses to the dis- 
tractor target are not as clear in Figure 4A, but it still can be 
noticed that this neuron’s rate was higher following the distrac- 
tor target appearance at the preferred locations (90° and 180°) 
compared to the non-preferred locations (0° and 270°). 

Figure5 illustrates, using the same neuron as shown in 
Figure 4, that the distractor target influenced neuronal pat- 
terns even when the monkey moved directly to the true tar- 
get (Figures 5B,C, left panels) as it did in the no-switch trials 
(Figure 5A). 

Figure 6 shows neuronal activity patterns for the entire pop- 
ulation of M1 neurons recorded in monkey M. The format is 
similar to Figure 4 with the difference being that target locations 
were ranked for each neuron according to the firing rate exhibited 
for each location. Trials without distracting targets (no-switch tri- 
als) were used to rank directions into the first preferred direction 
(Pref 1), second preferred direction (Pref 2) and so on for each 
neuron. As in Figure 4, no-switch trials are shown on the diag- 
onal (denoted by gray boxes). The off-diagonal panels show the 
switch trials with 250 ms SSDs. Figure 6B shows average PETHs 
for each SSD. Population PETHs (Figure 6A) and their averages 
(Figure 6B) indicate that M1 firing rates reflected both the dis- 
tractor (modulations for different columns of panels with the 
same row) and the true (different rows within the same column) 
target locations. It can be also seen that each configuration of the 
distractor and true target locations was associated with a unique 
pattern of population activity. Here, as in Figure 4, the initial 
component of the response is modulated across panel columns 
(i.e., representation of the distractor target), and the late compo- 
nent is modulated across panel rows (representation of the true 
target). Average PETHs of Figure 6B indicate that the duration 
of the distractor was clearly reflected by the population activity— 
both as the average PETH shape and (e.g., bottom row of panels) 
and its amplitude (e.g., top row of panels). One can also notice the 
location of the true target was clearly reflected as average PETH 
amplitude for all conditions, and the location of the true target 
was better reflected by average PETHs for longer SSDs—both as 
PETH shape and amplitude. Because of these differences between 
PETHs for different conditions, we were able to extract informa- 
tion about target locations from neuronal ensemble activity using 
a discrete classifier. 

The results from Figure 6 are additionally clarified in Figure 7 
that shows the evolution of neuronal rates using 4 x 4 dia- 
grams. In the diagrams, vertical and horizontal bands correspond 
to neuronal tuning to the distractor and true target location, 
respectively. The diagrams are shown for different time with 
respect to true target onset (left to right) and for different SSDs 
(Figures 7A—C). The most prominent result seen in all SSD 
groups is the emergence of the true target location (bottom hor- 
izontal band of the 4 x 4 grid) as the strongest modulator of 
population MFR. The effect of the distractor is weaker. Early in 
the trial, most clearly in the 0 to 200 ms epochs, the distrac- 
tor location is the primary modulator of neural activity. This is 
especially prominent when following a long SSD (Figure 7C). 
Notably, the effect of the distractor continues well into the true 
target presentation epoch. This is clear from the regression coeffi- 
cients shown above each panel indicating the strength of the role 
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FIGURE 5 | PETH of a single M1 neuron during specific transitions 
after 250 ms SSD. (A) Neural activity from no-switch trials separated 
by target location shows directional preferences with PETH and 
single-trial raster plots (below PETH) aligned on distractor target onset. 
Inset shows cursor trajectory from trials to the specified target. 

(B) Among switch trials, PETH and raster plots generated from trials 
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with distractor in one of the neuron’s preferred direction (90°) and 

the true target in a non-preferred direction (0°). Data from both direct 
(left) and distracted (right) shown, with inset showing cursor traces. 

(C) Same as (B) except data drawn from trials where both the distractor 
and true target are in preferred directions (90° and 180°, 

respectively). 


of true target and distractor target preference ranking on MFR 
(see Methods). At short SSDs (Figure 7A), the distractor coeffi- 
cient never reaches the level of significance (p > 0.05), however, 
the true target is strongly represented. At both 150 ms and 250 ms 


SSDs (Figures 7B,C), the distractor contributes a smaller, but still 
significant amount to the MFR. The MFR, even 400 ms after the 
distractor disappears, is still influenced by both targets as seen 
by the lower left corner triangle seen in the 400-500 ms epoch in 
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FIGURE 6 | Population activity from M1 neurons in Monkey M 
aligned on distractor target onset. (A) Normalized firing rate for each 
cell and each pairwise combination of distractor and true targets 
shown for the SSD of 250 ms condition. Amplitude of firing shown by 
color scale (on right) interpreted as the z-score. Position within the 

4 x 4 grid for each cell determined separately according to directional 


preference order. Data along the diagonal is from no-switch trials. 
(B) Population mean PETH for each of the three SSD conditions. 
Time-series data averaged across neurons within a specific condition 
(within one box on the 4 x 4 grid) organized by neuron directional 
preference ranking. Along the diagonal is mean population PETH for 
no-switch trials. 


Figures 7B,C. More generally, a transition occurs from distractor 
representation to true target representation. 

Average PETHs were also used to analyze the difference in 
neuronal patterns between correctly and incorrectly performed 


switch trials. The strategy for separating trials into these two 
groups is described in the Methods. Average PETHs were calcu- 
lated in the following steps. First, PETHs for each neuron were 
computed for each of 12 possible combinations of distractor and 
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true target locations. This computation was performed separately 
for correctly and incorrectly performed trials. Then, PETHs for 
correct trials were grouped together for all combinations and all 
neurons, and an overall average PETH was calculated. An average 
PETH for incorrect trials was calculated, as well. This computa- 
tion assured that the averages were not biased by the proportions 
of correct and incorrect trials for different conditions. We chose 
to average across all conditions because the differences for such 
averages were not specific to certain combinations of target direc- 
tions. As seen in Figure 8, the average PETHs differed depending 
on whether the monkeys successfully switched to the true target. 
More specifically, trials where the monkeys switched to the true 
target had lower initial slopes of firing rates than trials where the 


monkeys failed to reach to the true target. This effect was observed 
in both monkeys, both in M1 and S1 neurons. In monkey M, 
the neural activity in both M1 and S1 clearly rose before the tar- 
get switch and more steeply in trials where the monkey failed to 
switch to the true target. In monkey N, the difference in FR slopes 
was more subtle and occurred later than in monkey M, which 
was likely related to the more variable behavior of that monkey 
(Figure 2). 

Among rewarded trials, we analyzed neuronal representation 
of the target switch (Figure9). To do this, the trial-averaged 
PETH for switch trials was directly compared with the PETH of 
no-switch trials (see Methods). As the metric for the neuronal 
representation of the switch, we used RMS differences between 
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FIGURE 8 | Average population PETH differs depending on 

whether a switch is successfully made. Single neuron activity was 
averaged across all trials where the distractor was presented. Each such 
trial was categorized as either a successful switch (solid line), where the 
true target was reached, or unsuccessful (dashed line) when the monkey 


Monkey switched to new target 
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failed to reach the target and the trial was categorized as distracted 
(see Methods). Single neuron PETHs aligned on the true target onset, 
for both behavioral categories were then normalized to z-score and 
averaged across the population of monkey M M1 (A), monkey M S1 
(B), and monkey N M1 (C). 


PETHs for the trials with no target switches (diagonal data in 
Figures 4 and 6) and the trials with target switches. Figure 9A 
shows RMS differences averaged across all possible locations of 
the distractor and true targets for each individual neuron, and 
Figure 9B shows the result of additional averaging across neurons 
in particular cortical areas. These averages depict the intervals 
when neuronal activity started to reflect the target switch from 
its initial position. 

It is clear from Figures 9A and B that neurons represented 
target switch in M1 and S1 of both monkeys. Monkey M M1 pop- 
ulation represented the timing of target switch for all tested SSDs, 
as evident from the latencies of the average curves (Figure 9B, 
left). S1 population of monkey M resolved the timing of the 
switch at 250 ms from the switches at 50 and 150 ms. The dif- 
ference in switch timing was less clear in monkey N. Because of 
these representations of both the distracting and true targets by 
M1 and S1 neurons, we were able to extract target information 
from neuronal population activity. 


EXTRACTION OF TARGET POSITION WITH LDA CLASSIFIER 

An LDA classifier extracted the position of distractor 
(Figure 10A) and true (Figure 10B) targets from ensemble 
activity of M1 and S1 neurons. In the analysis depicted in 
Figure 10, predictions of target position were obtained from a 
short (150ms) window slid along the task epoch. Behavioral 
trials were aligned on the distractor target onset in this analysis, 
and the LDA classifier was trained anew for each window 
position. Prediction accuracy was calculated as a fraction of 
correctly predicted target locations. The analysis was performed 
separately for 50, 150, 250 ms SSDs (red, green, and cyan traces, 
respectively) and no-switch trials (black trace). The LDA classifier 
revealed the changes in the representation of the distractor and 
true target locations as a function time. Note that the true target 


could be decoded with high accuracy despite the appearance of 
a distracting target. This accuracy approached 90% correct in 
monkey M and could be decoded nearly as fast with a distractor 
as without a distractor (Figure 10B). With longer SSDs, the 
ability to decode the true target remained similar but occurred 
at a longer latency. Such good decoding of the true target is 
not surprising given that the monkeys’ overt behavior consisted 
of reaching movements to the true target. Future work should 
probe this decoding under real-time BMI control without overt 
behavior. The LDA model used for Figure 10 included training 
data from all SSD conditions, as described in the Methods. 
We found that limiting training data to only no-switch trials 
reduced the fraction correct over all SSD groups and predicted 
parameters in monkey M M1 by 19.9% (p < 0.01, paired f-test), 
monkey M S1 by 6.8% (p < 0.01), and monkey N M1 by 4.2% 
(p < 0.01). 

Consistent with the results of Figure 7, we observed that the 
representations of both the distractor and true targets outlasted 
the duration of target presence on the screen. In particular, the 
representation of the distractor lasted much longer than SSD 
(Figure 10A). Monkey M M1 ensemble provided the best predic- 
tions of the distractor target location, as it detected SSDs as short 
as 50 ms. SSDs of 150 and 250 ms were clearly represented by the 
M1 and S1 ensembles in monkey M and the M1 ensemble in 
monkey N. The duration of the distractor location representation 
increased with the increased SSD, and for all SSDs it extended well 
into the true target epoch when the distractor target was turned 
off. The onset of the representation of the true target matched 
the true target onset, and the prediction accuracy was higher for 
the true target than for the distractor target in all cases. The peak 
LDA predictions for distractor location from the monkey M M1 
ensemble were 20.2% more accurate than those for monkey N 
M1 and 34.3% more accurate when predicting the true target. 
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neuronal populations. (A) RMS difference computed for each cell (see 
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onset. Data shown separately for three SSD groups, with rows 1-3 the cell 
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each panel, the rows of the color plot indicate one single cell and the row 
height was fixed in all three cell groups (the three columns). The panel 
size is thus a reflection of the number of neurons in this population. 

(B) The population average RMS difference was computed from each 
panel in (A). 


To clarify whether this was the effect of different ensemble sizes, 
we repeated the analysis of Figures 10A,B for monkey M and 
N M1 populations using equally-sized subsets of each (n = 35 
neurons; not shown in figure). We found that the prediction 
performance disparity in Figure 10A between the two monkeys 
became less pronounced (monkey M M1 now only 9.9% more 
accurate in predicting distractor location), but still existed, when 
using nm = 35 neurons for both. The difference between the two 
groups in Figure 10B remained approximately the same (monkey 
M M1 now 33.1% more accurate than monkey N M1 in pre- 
dicting true target location). From this we conclude that the size 
of the neuronal population was one contributing factor to LDA 
performance, but there were other factors as well, such as more 
erratic performance of monkey N and characteristics of recorded 
neuronal populations. 

Since the predictions of the distractor target location by M1 
and S1 ensembles could simply reflect the fact that in a portion 


of trials monkeys initiated movements to that target, we sepa- 
rately analyzed the trials in which the monkeys moved directly 
toward the true target (direct trials; Figures 11A,C,D) and those 
in which the presence of the distractor affected the movement tra- 
jectory (distracted trials; Figures 11B,E,F). After the direct trials 
were separated, the predictions remained similar to those shown 
in Figure 10, indicating that cortical populations represented 
the distractor target even when the monkey did not produce 
movements toward that target. Curiously, monkey M M1 and 
S1 ensembles predicted the distractor target location even better 
when that monkey made straight movements to the true target. 
This was likely related to the predominance of such direct trials 
in the training data, resulting in a better prediction model. An 
opposite effect was observed for monkey N, presumably because 
it produced less direct movements. The predictions of the true 
target were similar for direct and distracted trials, and similar to 
the predictions shown in Figure 10. Note that the predictions of 
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the distractor location for the shortest, 50ms, SSD were much 
less accurate compared to the difference plots of Figure 9. This 
was because the LDA training data, unlike the Figure9 data, 
included all possible SSDs, as well as no-switch trials. The model 
had to generalize to all these conditions, which led to less accu- 
rate predictions for less represented cases. When 50 ms SSDs were 
analyzed separately by an LDA, the predictions improved (not 
shown). 

In addition to the distractor and true target location, we 
trained an LDA classifier to predict the target switch signal as a 
binary variable (Figure 12). Along the task interval, predictions 
of the distractor and true target locations were made concur- 
rently to serve as a temporal reference. As shown in Figure 12A, 
the strongest predictions made by LDA were for the true target 
in each case. Both monkey N and M M1 populations repre- 
sented the distracting target with approximately equal facility 
relative to the true target representation. Overall, the predictions 
were less robust that those predicted when separated by SSD 
(Figure 10) and behavior (Figure 11). This was a result of com- 
puting the fraction correct of all three SSD groups collectively, 
rather than separately. This caused the less strongly predicted 
50ms SSD trials to reduce the classifier performance overall. 
As shown in Figure 9, the temporal profile of neural activity is 
strongly dependent on the SSD ofa given trial. Target switch was 


moderately decoded in all three populations, each time with the 
peak occurring in the span between distractor and true target 
representations. The exact timing of the switch signal represen- 
tation is dependent on the SSD and thus the peaks that are 
present in Figure 12 represent approximate event epochs. The 
variation on when the target switch occurs relative to the dis- 
tractor onset—the time which all data is aligned to—is likely a 
contributing factor to the low fraction correct. If a single SSD 
were to be used, the switch event detection would likely occur in 
the SSD-dependent windows found in Figure 9B. The strongest 
representation of both the distractor and the switch event (note 
the scale difference) were obtained from monkey M M1 neurons. 
This is consistent with our previous data and strongly correlates 
with the higher number of quality recorded neural units in the 
Monkey M M1 population. Furthermore, the binary classifica- 
tion of switch vs. no-switch trial was evaluated in terms of the 
MCC (Figure 12B). In all three neuronal populations, the peak 
MCC for the switch detection occurred within 700 ms of the dis- 
tractor onset, although MCC begins to rise as early as 300 ms in 
monkey M M1. The performance of detecting the switch event 
was strongest among M1 cells, with peak MCC of 0.19 (monkey 
M) and 0.23 (monkey N), respectively. Thus, the neural basis for 
motor plan switching can itself be decoded from ensembles of M1 
and S1 neurons. 
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DECODING OF CURSOR AND TARGET POSITION USING WIENER FILTER 

We next utilized a continuous linear decoding algorithm, the 
Wiener filter, to predict cursor and true target position at a 10 Hz 
output rate throughout the session (Figure 13). A representative 


25 s window of predicted parameters, along with actual parameter 
values is shown in Figures 13A—D for Monkey M. Cursor X and 
Y were predicted with high correlation to actual movements in 
monkey M (X: R = 0.84; Y: R = 0.86) and moderate correlation 
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FIGURE 12 | Decoding of target location and switch/no-switch 
occurrence from neural activity during task interval, not separated by 
SSD. (A) Linear discriminant analysis predictions of location of distractor 
target (black) and second target (cyan) as well as switch/no-switch (red). Data 
aligned on distractor onset shown with vertical black line. Values shown are 
fraction correct prediction at each time step of the sliding window. Gray 
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in monkey N (X: R = 0.49; Y: R = 0.33). Computing the pre- 
dicted radial movement, 1, resulted in clear peaks indicating 
predicted reach events (Figure 13B). True target location was 
decoded very effectively (Figure 13C). To quantify this, predic- 
tions aligned on threshold crossing of r were made (see Methods). 
We found that the threshold crossing event occurred often 
within 700 ms after the true target appeared in both monkeys 
(Figures 13D,E). Performance of cursor and true target location 
predictions remained consistent for approximately 500 ms after 
threshold crossing, before declining. (Figures 13D,E, insets). In 
both monkeys, the cursor position was predicted highly effec- 
tively (up to 99%, monkey M; 59% monkey N). Prediction of 
true target position was strong from Monkey M (up to 59%), 
but was much weaker for monkey N (up to 26%). Chance level 
prediction in this case was 20% due to four potential target loca- 
tions and one condition where the target was not on the screen. 
The results from this continuous approach agree with the find- 
ings using a discrete LDA classifier. Furthermore, the analysis of 
Figure 13 provides evidence that the results of this study could be 
implemented into BMI systems to continuously extract intended 
reach locations. 


DISCUSSION 


In this study we examined M1 and S1 ensemble activity recorded 
in a motor task that required reprogramming of center-out reach- 
ing movements to visual targets. This was achieved by changing 
the target location in the midst of the RT period (Georgopoulos 
et al., 1981, 1983). We hypothesized that BMI decoding algo- 
rithms could dissociate representations of potential and selected 
motor targets from the activity of sensorimotor cortex ensembles. 


We found that locations of distracting targets presented shortly 
before the true targets of movements were indeed represented 
by M1 and S1 ensembles and could be extracted by an LDA 
classifier. The LDA results were recapitulated using a contin- 
uous Wiener filter which extracted cursor and target location. 
These results suggest that real-time BMI decoders could be 
implemented in the future to decode motor programming and 
decision making under the conditions of multiple potential 
choices. 

Despite the behavioral differences between the two monkeys 
in this study, as is common in primate studies, both helped 
to elucidate behavioral responses and the neural basis for tran- 
sient distractors. In our previous study (Ifft et al. 2011) we 
overtrained these monkeys to perform center-out movements 
with high accuracy when no distractor was used. In monkey N, 
the distractor markedly changed movement trajectories. Thus 
the distractor and true target locations were represented by both 
the overt behavior and cortical modulations. Monkey M was less 
distracted and the cortical effect of a change in motor plan could 
be studied, even when movements to the first target were wholly 
absent. 


SENSORIMOTOR CORTEX AND REPROGRAMMING MOVEMENTS 

Neural processes of motor program selection and cancellation has 
received much attention during the last two decades of research. 
The summary of this body of work suggests that different aspects 
of sensorimotor transformations that involve multiple potential 
choices are processed by multiple cortical and subcortical areas 
(Crammond and Kalaska, 1994; Shen and Alexander, 1997; Lee 
and Assad, 2003). Here we recorded ensemble activity in M1 
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and S1—the areas most closely reflecting the final motor out- 
put that results from decision making. Consistent with previous 
work (Alexander and Crutcher, 1990) we observed M1 activity 
that represented potential motor targets even when no movement 
was initiated toward those targets. This representation persisted 
well beyond distractor disappearance and the termination of this 
encoding coincided with the onset of the robust second (true) tar- 
get representation. Somewhat surprisingly, we observed moderate 
movement and pre-movement modulation in S1—an area whose 
primary function is commonly assumed to be related with sen- 
sory processing, but also known to be activated in advance of 
movements (Soso and Fetz, 1980; Nelson et al., 1991; Lebedev 
et al., 1994) and encode information about potential reach targets 
(Ifft et al., 2011). 

Here we cannot resolve whether representation of potential 
targets that we observed in M1 and S1 merely reflected inputs 
from associative areas that were the primary players in target 
selection (Thaler and Goodale, 2011) or M1 and S1 consti- 
tuted an integral part of a distributed network with less clearly 
defined hierarchy (Shen and Alexander, 1997; Hernandez et al., 
2010). Visuomotor information has been shown to be encoded 
by cortial visual processing networks in parietal (Kertzman 
et al., 1997; Wise et al., 1997; Baumann et al., 2009), premo- 
tor (Crammond and Kalaska, 1994; Lebedev and Wise, 2000; 
Cisek and Kalaska, 2002), and prefrontal (Genovesio et al., 2005; 


Lebedev et al., 2005) areas. These associative areas could act as 
filters of sensory information that is subsequently signaled to M1 
output areas. The exact mechanisms of interactions between non- 
primary and primary areas will have to be elucidated by future 
investigations. 

Our previous unpublished observations indicate that certain 
initial stages of target selection for a movement goal have to take 
place for target information to start to be represented in M1 and 
S1. In that experiment, animals had to deal with two targets that 
appeared on the screen simultaneously instead of in rapid succes- 
sion. One of the targets was large, and the other was small. The 
monkeys would be rewarded for reaching to either of the targets, 
but they typically selected the larger target because it was easier 
to hit with the cursor. In contrast to the results from our distrac- 
tor experiments reported here, in the unpublished study M1 and 
S1 neurons represented the non-chosen target in a much more 
subtle way, with less than 10% of recorded cells exhibiting any 
significant directional tuning to its location. This observation, in 
the context of the results of the present study, suggests that M1 
and S1 representation of movement direction is much stronger 
when the motor goal is chosen, even if only for several hundred 
milliseconds. 

Serial activation of M1 during motor sequences has been well- 
studied (Fu et al., 1995; Tanji, 2001) and the results of our study 
suggest that the manifestation of change-of-decision in the motor 
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cortex is a sequential, but somewhat overlapped representation of 
distinct motor plans. In other words, sensorimotor cortex repre- 
sents selected motor targets, but movements to those targets can 
still be canceled. Such movement cannot be canceled if M1 activ- 
ity is already elevated and has reached a certain motor initiation 
threshold (Figure 8) (Hanes and Schall, 1996; Wong-Lin et al., 
2010). 

A prominent model to describe the change-of-decision is a 
bounded form of the accumulator model (Vickers and Smith, 
1985), drift-diffusion model, or race model with criterion bound- 
aries for both initial decision and change of decision events 
(Resulaj et al., 2009). Applying this model to neurophysiology 
of sensorimotor neurons, one hypothesis would be that the fir- 
ing rate of a single neuron or entire neuronal populations would 
encode the degree of commitment to the specific motor plan. 
Lower levels of activity would elongate the decision window while 
additional evidence is accumulated, even if a different move- 
ment had been initiated. To address this hypothesis our study 
compared population activity between a subset of trials where 
the true target was successfully reached versus trials where an 
error in behavioral outcome was caused by the transient distrac- 
tor presence. Whether this lower population activity is causal 
to the behavioral differences is beyond the scope of this study. 
However, our results reinforce this model by showing lower 
initial population activity and more gradual FR slope between 
distractor target presentation and movement initiation on tri- 
als where the switch was successfully made (Figure 8). Such 
differences in population activity may provide intuitive under- 
standing for the ability to detect the switch/no-switch event 
using neural activity from single trials (Figure 12B) with high 
fidelity. 


DECODING MOTOR REPROGRAMMING 
Here we used a rather simple LDA decoder that extracted tar- 
get and target switch information from both cortical activity and 
the timing of the distractor target onset. This decoder was use- 
ful to describe the representation of targets by neuronal activity 
as a function of time. A practical decoder will have to extract 
target onset, as well. Our BMI approach added an interesting 
twist to our experiments because information extracted from dif- 
ferent parts of sensorimotor hierarchy could be used to retrain 
brain circuitry. For instance, learning a BMI task that involves 
extraction of target information may result in an enhanced repre- 
sentation of such information in M1. Additionally, non-primary 
areas should be considered as sources of information about mul- 
tiple potentials targets (Snyder et al., 1998; Cisek and Kalaska, 
2002, 2005) for a practical real-time decoder. With the cur- 
rent approach, we were able to extract the location of distractor 
targets from the primary sensorimotor cortical activity even if 
those targets were presented for a brief period of time (as short 
as 50ms) and if no movement was initiated to that target. It 
is important to emphasize that under this same condition, the 
true target to which the monkeys moved was also decoded very 
accurately. 

As the BMI field advances, practical, versatile neuroprosthet- 
ics based on BMI technology become a real possibility (Lebedev 
and Nicolelis, 2006; Nicolelis and Lebedev, 2009; Gilja et al., 


2011; Jackson and Fetz, 2011; Lebedev et al., 2011). The need 
for practical clinical applications that provide higher degree of 
freedom control (Velliste et al., 2008) and expanded decod- 
ing strategies (Zacksenhouse and Nemets, 2008) will drive BMI 
research to expand into more complex motor programs. Naturally 
enacted movements require the flexibility to rapidly modify 
upcoming motor plans. Such a behavior capability was reflected 
in the neuronal data we collected in the present study. The 
ability to decode such changes has critical implications for 
not only accuracy but also safety in the execution of every- 
day movements by a prosthetic device controlled by brain 
activity. 

Our present experimental approach, based on a discrete rather 
than continuous decoder, adds to previous literature where sim- 
ilar ideas were evaluated under the framework of a potential 
cognitive neuroprosthetic (Musallam et al., 2004; Pesaran et al., 
2006). A cognitive neuroprosthetic extracts from brain activ- 
ity information that is different from motor execution signals 
and utilizes it to improve the performance. For example, a 
high-performance BMI proposed by Santhanam et al. (2006) 
extracted target location from delay-period activity recorded in 
dorsal premotor cortex and thereby obtained information trans- 
fer rate of up to 6.5 bits per second. Additional improvements 
may come from hybrid BMI designs that utilize both single-unit 
recordings and local field potentials (LFPs). Thus, Hwang and 
Andersen (Hwang and Andersen, 2009) decoded movement onset 
from LFPs while decoding movement direction from single-unit 
activity. 

Hasegawa et al. (Hasegawa et al., 2006, 2009) implemented 
decoding algorithms that served a similar purpose that we 
describe here. They decoded go/no go decisions from the activ- 
ity of 2-5 neurons recorded in monkey superior colliculus and 
were able to extract multidimensional decisions (e.g., go/no go 
for two potential movement directions). The information was 
accessed approximately 150 ms after cue onset, which is consis- 
tent with our present results and the results of Santhanam et al. 
(2006). Given a high interest to neurophysiological mechanisms 
of response inhibition (Hanes and Schall, 1995; Pare and Hanes, 
2003; Chen et al., 2010; Scangos and Stuphorn, 2010; Mirabella 
et al., 2011), it is reasonable to expect that BMIs that extract 
response inhibition and response reprogramming information 
will continue to develop. 


VERSATILE BMIs OF THE FUTURE 

The original conception of BMI systems strive to mimic normal 
functions of the brain as closely as possible (Nicolelis, 2001). The 
approach that we propose here can be generally characterized as 
a BMI with impulsivity control. Impulsivity is a person’s inabil- 
ity to inhibit unwanted actions (Basar et al., 2010; Kim and Lee, 
2011). Prefrontal mechanisms are normally responsible for such 
inhibition in primates (Miller, 2000; Krawczyk, 2002; Kim and 
Lee, 2011). It is conceivable that practical BMIs of the future will 
need an inhibition control module to operate properly. Moreover, 
such a module may become one of the essential elements of 
the design. It may not only examine potential actions and select 
those that fit the context and are wanted by the user, but also 
set the limits to volitional control. In the past, we have already 
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proposed that such an optimal design may be based on a shared- 
control BMI, ie., one that gives the user control over higher- 
order goals and delegates lower-order controls to the robotic 
controller (Kim et al., 2006). A prominent role of prefrontal 
cortex is executive function, such as the one required for inhi- 
bition of potential actions. Future work could seek to exploit 
the multiple levels of control within the brain to not only recre- 
ate naturalistic movements, but at the same time streamline the 
transitions and selections from the many possible behavioral 
outcomes. Certainly this goal is challenging, but we remain opti- 
mistic in light of recent developments in the fast growing field of 


neuroprosthetics. 
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The brain representation of the body, called the body schema, is 
susceptible to plasticity. For instance, subjects experiencing a rub- 
ber hand illusion develop a sense of ownership of a mannequin 
hand when they view it being touched while tactile stimuli are 
simultaneously applied to their own hand. Here, the cortical basis 
of such an embodiment was investigated through concurrent re- 
cordings from primary somatosensory (i.e., $1) and motor (i.e., M1) 
cortical neuronal ensembles while two monkeys observed an 
avatar arm being touched by a virtual ball. Following a period 
when virtual touches occurred synchronously with physical brushes 
of the monkeys’ arms, neurons in $1 and M1 started to respond to 
virtual touches applied alone. Responses to virtual touch occurred 
50 to 70 ms later than to physical touch, consistent with the in- 
volvement of polysynaptic pathways linking the visual cortex to 
S1 and M1. We propose that $1 and M1 contribute to the rubber 
hand illusion and that, by taking advantage of plasticity in these 
areas, patients may assimilate neuroprosthetic limbs as parts of 
their body schema. 


multielectrode recordings | cortical plasticity 


[: the early 1900s, Head and Holmes coined the concept of the 
“body schema” to describe the spatial model of the body that 
the brain builds based on sensory inputs from the skin, joints, and 
muscles, as well as visual and auditory signals (1). Numerous 
studies since then have explored different aspects of the body 
schema (2-6), particularly the role of cortical areas (7, 8). The 
accumulated literature indicates that the body schema is plastic 
and can even incorporate artificial tools (5, 9, 10). A striking ex- 
ample of body schema plasticity is provided by the rubber hand 
illusion (RHI), in which subjects start to perceive a mannequin 
hand as their own after their real hand, hidden from sight, and the 
mannequin hand are repeatedly touched simultaneously (11-13). 
Subjects do not perceive a third limb, but report a shift in position 
sense from the real arm to the fake one (11-14), and there is even 
a decrease in skin temperature of the real arm (15). Incorporation 
of artificial limbs into the body schema began to be further ex- 
plored with the advancement of brain machine interfaces (BMIs), 
hybrid systems that connect the brain with external devices (16- 
19). Here, we recorded cortical ensemble activity in monkeys 
exposed to the paradigm that elicits RHI in humans (11-14, 20). 


Results 

Monkeys M and N were chronically implanted with microwire 
arrays in the primary motor (i.e., M1) and somatosensory (i.e., S1) 
cortical neuronal ensembles. They observed a 3D image of a vir- 
tual arm (i.e., an avatar arm) being touched by a virtual ball on an 
LCD screen while a robot slid a physical brush through the skin of 
their real arms (Fig. 1.A and B). The virtual touch (V) and physical 
touch (P) were synchronous or asynchronous (Fig. 1C). In a subset 
of trials, virtual brushing occurred alone (i.e., Vonly). 


Excitatory and Inhibitory Responses to Physical Touch. Experiments 
with monkey M started with epoch 1 (E1) that consisted of 100 


www.pnas.org/cgi/doi/10.1073/pnas. 1308459110 


Vonly trials, followed by epoch 2 (E2) in which 100 Vonly trials 
were intermingled with 900 virtual plus physical (V+P) trials 
(Fig. 1D). For monkey N, experiments contained only E2. Two 
synchronous V+P (V+P sync) sessions and one asynchronous 
V+P (V+P async) session were conducted on separate days for 
each monkey. Neurons in S1 and M1 clearly responded in V+P 
syne trials (Figs. 2 and 3), as confirmed by cumulative sum 
analysis (21, 22). The excitatory responses prevailed (122 excit- 
atory vs. 8 inhibitory responses for both monkeys; Table S1). For 
monkey M, neuronal responses occurred throughout the brush- 
ing period because its receptive fields (RFs) were comparable in 
size with the brushed area on the forearm and were distributed 
over that area. For monkey N, responses were prominent toward 
the end of the brushing period when the brush entered the 
smaller RFs on the base of the index finger. 

In monkey M, 62 neurons were sampled in S1 and 82 were 
sampled in M1. In V+P sync trials, 73% of S1 neurons (n = 44) 
had excitatory responses, none had inhibitory responses, and 
27% (n = 18) were not modulated (Table S1). In M1, 79% (n = 
64) of neurons were excited, 6% (n = 5) were inhibited, and 16% 
(n = 13) were not modulated. In monkey N, 15 neurons were 
sampled in S1 and 34 were sampled in M1. In V+P trials, 33% of 
the S1 neurons (7 = 5) were excited, no neurons had inhibitory 
responses, and 67% (n = 10) did not respond (Table S1). In M1, 
26% neurons (1 = 9) were excited, 9% (n = 3) were inhibited, 
and 65% (n = 22) did not respond. 


Responses to Virtual Touch. A significant number of S1 and M1 
neurons with V+P responses started to respond in Vonly trials 
during E2 (Figs. 2 and 3 and Tables S1 and S2). In monkey M, 
such Vonly responses occurred in 50% (n = 31) of the total of 62 
S1 neurons (all excited, none inhibited) and 70% (n = 57) of 82 
M1 neurons (51 excited vs. 6 inhibited; Table S1). In monkey N, 
Vonly responses occurred in 40% (n = 6) of 15 S1 neurons (five 
excited vs. one inhibited) and 38% (n = 13) of 34 M1 neurons 
(eight excited vs. five inhibited). Fig. 24, Left, illustrates peri- 
stimulus time histograms (PSTHs) for all individual trials of 
one S1 and two M1 neurons from monkey M. Arm electro- 
myograms (EMGs) were sampled simultaneously (Fig. S1). 
These neurons did not have Vonly responses during El, but 
responded in both V+P sync and Vonly trials during E2. One 
neuron (Fig. 24, Center) was inhibited, whereas the other two 
were excited. Fig. 2A, Right, shows two monkey N neurons with 
excitatory responses during both V+P synch and Vonly trials of 
E2, and Fig. 2B depicts average PSTHs with excitatory responses 
for both monkeys. 
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Experimental paradigm. (A) The monkey's arms were restrained underneath a computer screen. Three-dimensional arms were shown on the screen, 


with the size and position approximating the arms. Physical touches were applied with a brush mounted on a robot. (B) Skin locations to which brushing was 
applied. (C) Combinations of virtual and physical touch: V+P sync (Top), Vonly (Middle), and V+P async (Bottom). (D) Trial sequence: E1 consisted of 100 Vonly 
trials and E2 consisted of 100 Vonly trials intermingled among 900 V+P trials. Synchronous paradigm is on the left (yellow background); asynchronous is on 


the right (gray). 


Cortical responses during Vonly trials could not be explained 
by arm movements, as shown by EMG controls (Fig. $1). EMGs 
occurred sporadically, but were not time locked to the stimuli 
(P > 0.5, Wilcoxon test) and decreased as the session continued, 
despite the S1 and M1 responses remaining high (Fig. 2.4). 

Fig. 3 depicts population PSTHs, their averages and cumula- 
tive sums for neurons with significant V+P responses. Vonly 
responses occurred only during E2 of the synchronous paradigm. 


Response Magnitude. For the neurons shown in Fig. 3, neuronal 
rates were measured before and after the stimuli for different 
monkeys, areas, and paradigms (Table S2). Response magnitude 
was quantified as rate change from the baseline period before 
stimulus application (AR). AR was expressed in spikes per sec- 
ond and normalized units, i.e., z-scores. In monkey M, during the 
synchronous paradigm, AR in Vonly trials was approximately 
half that of AR in V+P trials (Table S2). S1 neurons with ex- 
citatory responses increased their firing rate from the baseline 
34.9 + 2.7 spikes per second to 44.8 + 3.5 spikes per second in 


W 


$1 neurons 
Monkey N V+P 


Monkey M 
S1 neuron M1 neuron M1 neuron 


Monkey M 
Vonly(E2) ve 


V+P sync trials (AR of 9.9 + 2.7 spikes per second) and from 
34.9 + 2.7 spikes per second to 39.0 + 2.9 (AR of 4.2 + 2.7 spikes 
per second) in Vonly trials of E2. M1 neurons increased rate 
from 17.7 + 2.0 spikes per second to 29.6 + 2.9 (AR of 12.0 + 2.0 
spikes per second) in V+P sync trials and from 17.1 + 1.9 spikes 
per second to 22.6 + 2.4 (AR of 5.5 + 1.9 spikes per second) in 
Vonly trials of E2. Neither S1 nor M1 neurons responded in 
Vonly trials of El (AR of 0.7 + 3.0 and 1.1 + 2.0 spikes per 
second, respectively). In monkey N with only E2 data, Vonly 
responses were of the same magnitude as V+P sync responses, 
which may indicate a stronger RHI in that monkey compared 
with monkey M (Table S2). S1 neurons with excitatory responses 
increased rates from 16.0 + 1.4 spikes per second to 23.5 + 4.0 
(AR of 7.5 + 1.4 spikes per second) in V+P sync trials and from 
15.8 + 1.5 spikes per second to 22.2 + 3.4 (AR of 6.5 + 1.5 spikes 
per second) in Vonly trials. Excited neurons in M1 increased 
rates from 25.8 + 6.1 spikes per second to 35.0 + 6.2 (AR of 9.2 + 
6.1 spikes per second) in V+P sync trials and from 24.9 + 5.8 
spikes per second to 33.7 + 6.2 (AR of 8.9 + 5.8 spikes per 
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sent baseline rates. 
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Fig. 3. Populations responses in monkey M and N. (A, B, F, and G) PSTHs for responses of $1 and M1 neurons for synchronous (shaded yellow) and asyn- 
chronous (gray) sessions. Firing rates are normalized to represent z-scores. In all plots, only neurons with significant excitatory responses to physical touch are 
shown. The neurons are shown in ranked order from the weakest (Top) to the strongest (Bottom) responses. Color plots show the same set of neurons for 
Vonly from E1 (Top), Vonly from E2 (Middle), and V+P (Bottom). Vertical green lines represent onsets and offsets of the stimuli. Average PSTHs for the 
population are indicated by black lines superimposed on the color plots. Asterisks indicate significant differences between response rate and baseline by 
Wilcoxon text: *P < 0.05, **P < 0.01, and ***P < 0.001. (C, D, H, and /) Average and cumulative PSTHs for the population for monkey M (C and D) and monkey 
N (H and /). Pink lines represent the mean and SD. (E and J) Distribution histograms for the difference in response onsets between V+P sync trials and Vonly 
trials for $1 (Upper) and M1 (Lower) neurons. Positive values indicate that V+P sync responses occurred earlier than Vonly responses. Red dashed line is the 
Gaussian fit. Arrows mark mean values. Mean and SEM are indicated. 


second) in Vonly trials. Vonly responses, measured in normal- monkey M (P < 0.001 for S1 and P < 0.001 for M1, ¢ test; Fig. 4.4 
ized units, were significantly higher during E2 than during El for and B, Left) and for monkey N (P < 0.001 for S1 and M1; Fig. 4 
both S1 (P < 0.001, Wilcoxon test; Fig. 4£) and M1 (P < 0.001; ~—-C and D, Left). No such correlation was found between Vonly 
Fig. 4F). Furthermore, the magnitude of Vonly responses during responses during El and V+P sync responses during E2 in 
E2 was correlated with the magnitude of V+P sync responses for monkey M (P > 0.1 for S1 and M1, ¢ test; Fig. 4.A and B, Left). 
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Fig. 4. Analysis of responses to virtual and physical touch. (A-D) Scatter 
plots for the correlation analysis of responses in different conditions. Each 
dot represents an individual neuron. V+P response magnitude (in spikes per 
second) is plotted against Vonly response magnitude. The results of linear 
regression are indicated by dashed lines, and corresponding functions are 
shown above the figures. (E-H) Bar plots representing mean and SEM for 
response amplitude in Vonly trials across cortical areas, monkeys, epochs 
(i.e., E1 and E2), and experimental conditions. Synchronous sessions are in- 
dicated by yellow background, asynchronous by gray background. Firing 
rates are normalized to z-scores. Only the neurons with significant excitatory 
responses to physical touch were considered. Asterisks indicate significant 
differences by Wilcoxon test: *P < 0.05, **P < 0.01, and ***P < 0.001. 


Asynchronous Physical and Virtual Touch. Vonly responses did not 
emerge under the asynchronous paradigm (Fig. 4 A—D and 
Tables S1 and S2). In monkey M, they occurred in only 4 of 62 S1 
neurons (6%; one excited, three inhibited) and 4 of 82 M1 
neurons (5%; two excited vs. two inhibited; Table S1). In monkey 
N, Vonly responses occurred in 2 of 15 S1 neurons (12%; one vs. 
one) and 2 of 34 M1 neurons (6%; one vs. one). Additionally, 
V+P async responses in monkey N were somewhat lower than 
V+P sync responses in the same monkey (Fig. 3 F—/). Whether 
this difference was just a statistical variability will have to be 
tested in future studies. An absence of Vonly responses under 
the asynchronous paradigm is clear from the average and cu- 
mulative PSTHs (Fig. 3 C, D, H and J). 

Additionally, the analyses of response magnitude and corre- 
lation (Fig. 4 E-H) indicate clear differences between synchro- 
nous and asynchronous sessions. Vonly responses during E2 
were significantly higher under the synchronous paradigm than 
under the asynchronous one for both monkey M (Fig. 4 E and F) 
and monkey N (Fig. 4 G and #). Finally, no correlation was 
found between Vonly and V+P async responses during E2 for 
either monkey (P > 0.1, ¢ test; Fig. 4. A—D). 

For the asynchronous sessions, we investigated whether the 
time lag between the virtual and physical touches had an effect 
on cortical responses. These analyses did not reveal any sig- 
nificant effect. 
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Response Latencies. For monkey M, the Vonly responses of S1 
neurons lagged the V+P sync ones by 67 + 3.5 ms (mean + 
SEM). The lag was 55 + 3.8 ms for M1 neurons (Fig. 3£). For 
monkey N, the lag was 53 + 20 ms for S1 and 60 + 12 ms for M1 
(Fig. 3/). For monkey M that had a clear response to the onset of 
brushing, response latency to the physical touch was 30 + 7 ms 
for S1 neurons and 25 ms + 10 ms for M1 neurons (not signifi- 
cantly different). Thus, V+P sync responses occurred ~30 ms 
after the stimulus, and Vonly responses were 90 ms after the 
stimulus. 


Discussion 


Our experiments showed that, following synchronous V+P stim- 
ulation, neurons in $1 and M1 start to respond in a matter of 
minutes to virtual stimuli delivered to an avatar arm alone. This 
time course is consistent with the development of the sense of 
ownership of an artificial limb examined in humans (11-13). 
Although our monkeys did not report their perceptions during 
virtual touches, the presence of cortical responses to virtual 
stimulation supports our assumption that a perceptual illusion 
was elicited in these nonhuman primate subjects, and that it was 
similar to the RHI in humans. Additional evidence indicates that 
the monkeys could treat the avatar as being an integral part of 
their bodies (Fig. $2). Indeed, the level of realism of the avatar 
was high enough for monkeys to treat it as a conspecific animal 
(Fig. S2.A—C). In our previous experiments, the same avatar arm 
served as the BMI actuator that performed active tactile explo- 
ration of virtual objects (18). Moreover, M1 and S1 neurons were 
attuned to the avatar movements when monkeys passively ob- 
served it move on the screen (Fig. S2D), and the avatar trajectory 
could be derived from cortical activity (Fig. S2E). 

On the whole, these results suggest that the avatar was realistic 
enough to become an equivalent of a “rubber hand” for our 
monkeys. Accordingly, we propose that the occurrence of sen- 
sory evoked responses in S1 and M1 neurons to virtual touches 
of the avatar may contribute to the experiences of the RHI. 
Future studies will have to address whether these S1 and M1 
neuronal firing modulations are essential for the illusion or if 
they are secondary to the activity of associative areas, for ex- 
ample activity of bimodal neurons in premotor and posterior 
parietal areas previously shown to play a role in the RHI in 
humans (12). 

Several previous studies reported visually evoked responses in 
S1. Activation of somatosensory cortical areas has been reported 
in human fMRI (2, 4, 5) and EEG (8) studies in which subjects 
observed images of body parts being touched. Such cortical 
activations are often interpreted as mirror responses to an ob- 
servation of actions (5). Mirror response is an unlikely expla- 
nation for our present results because simply viewing virtual 
touches without synchronous physical stimulation was insufficient 
for visually evoked responses to develop. $1 modulations to visual 
stimuli that differs from mirror responses previously have been 
reported in monkeys trained to perform a visuohaptic task that 
required short-term retention of tactile information (23). 

A somewhat surprising finding was the observation of prom- 
inent V+P and Vonly responses not only in S1, but also in M1. 
These responses could not be explained by overt movements 
because there were no accompanying EMG modulations. Our 
additional experiments showed that S1 and M1 responded to 
air-puff stimulation of the skin, and these responses were also 
unrelated to EMGs (Fig. S3). In our opinion, the presence of 
somatosensory and visual responses in M1 indicates that the 
cortical mediation of RHI is widely distributed throughout the 
frontoparietal circuit. The rich interconnections between M1 and 
S1, as well as their connections with the adjacent areas in the 
frontal and parietal lobes, provide a neuroanatomical substrate 
for such distributed representation (24). The absence of strict 
segregation of physiological functions between somatosensory 
and motor areas has been postulated since the mid-1900s (25). 
Several studies that used multielectrode recordings in rodents 
and primates have provided further support to this notion (26, 
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27). Our recent BMI studies suggest that individual S1 and M1 
cortical neurons may contribute to multiple computations si- 
multaneously (28). Moreover, both multielectrode recordings 
(29) and optical imaging (30) have revealed a highly distributed 
and dynamic representation of multiple information streams in 
the cortex. 

Although our data do not pinpoint the source of visually driven 
responses in S1 and M1 neurons, the long response latencies 
suggest that such responses may emerge as a result of poly- 
synaptic pathways linking visual cortical areas to the sensori- 
motor cortex (24). As evidence for direct projections from visual 
areas to S1 and M1 in monkeys is lacking to date, the most likely 
candidates for the source of such inputs are posterior parietal 
and premotor areas, which are one synapse away from S1 and 
M1. Traditionally, S1 and M1 have been associated with primary 
processing of sensory and motor signals, whereas posterior pa- 
rietal and premotor areas have been considered multimodal, 
integrative areas where the body schema is constructed. How- 
ever, in a highly interconnected frontoparietal cortical circuitry, 
it is practically impossible to maintain information confined to 
a single area. Therefore, it is not so surprising that the bimodally 
driven S1 and M1 neurons resemble equivalent neuronal re- 
sponses reported by Graziano et al. in the premotor cortex and 
posterior parietal cortex (31, 32). 

Previously, posterior parietal and premotor neurons were 
shown to generate anticipatory activity preceding the tactile 
responses during active touch (33). That study suggested that 
predictive and planning components of active touch were pro- 
cessed mostly in the posterior parietal and premotor areas, whereas 
S1 and M1 handled performance and feedback. However, this 
functional dissociation was not absolute. Furthermore, S1 neu- 
rons in rats exhibit anticipatory activity before whisker contact 
with an object (34). A study by Avillac et al. (35) argued that 
ventral intraparietal cortex represents visual and tactile infor- 
mation through a process of predictive multisensory integration. 
In theory, similar anticipatory mechanisms could have played 
a role in our study. However, we did not observe considerable 
activity preceding the skin touch. For instance, the Vonly re- 
sponses were rather late, as they lagged the V+P responses by 50 
to 70 ms. This lack of anticipatory firing may reflect the fact that 
our monkeys were not performing an active tactile discrimina- 
tion. Whereas anticipatory activity is unlikely, a more plausible 
hypothesis is that S1 neurons were conditioned by V+P sync 
trials to respond to the virtual touches. Such interpretation would 
still be consistent with the common representation of visual and 
tactile information suggested by Avillac et al. (35). 

Our present study has an implication for the future design of 
neuroprosthetic devices controlled by BMIs (17) by suggesting 
that rehabilitation training paradigms, which take advantage of 
cross-modal cortical plasticity observed here, will likely not only 
enhance the patients’ proficiency in the use of neuroprostheses, 
but also provide them with a renewed sense of ownership of 
these devices. 


Methods 


Subjects and Implants. All animal procedures were performed in accordance 
with the National Research Council’s Guide for the Care and Use of Labo- 
ratory Animals and were approved by the Duke University Institutional 
Animal Care and Use Committee. Two adult rhesus macaque monkeys 
(Macaca mulatta) were implanted with 96-microwire arrays constructed of 
stainless steel 304 (16-18). Within each array, microwires were grouped in 
two four-by-four grids each consisting of 16 electrode triplets. The separa- 
tion between adjacent triplets was 1 mm. The electrodes in each triplet had 
three different lengths, increasing in 300-4m steps. The penetration depth 
of each triplet was adjusted with a miniature screw. The longest electrode 
in each triplet penetrated to a depth of 2 mm as measured from the cortical 
surface. Each hemisphere received two arrays: one in the upper-limb rep- 
resentation area and one in the lower-limb representation area. These 
arrays sampled neurons in both M1 and S1. As the dura matter was opened 
during the surgery, all cortical surface features were clearly identified be- 
fore implantation. We recorded from the right-hemisphere arm arrays in 
each monkey. 


Shokur et al. 


RFs and Identification of Recording Sites. We recorded from the right cerebral 
hemisphere while delivering tactile/visual stimuli to the left side of the body 
and/or avatar. Physical brushing was applied to the forearm for monkey M 
and to the index finger for monkey N (Fig. 1B). Those were locations where 
the RFs of most $1 neurons were found after an examination that involved 
manually touching the skin with a cotton swab. S1 neurons had sensitive RFs 
clearly localizable on the animal's arm or hand. The $1 somatotopic maps 
were in good correspondence with previous studies. Localization of RFs was 
not as clear for M1 neurons as for S1 neurons. Passive movements of the 
fingers, hand, and arm were typically effective to activate M1 neurons. 

An additional test that we conducted to physiologically identify M1 vs. $1 
was an application of intracortical microstimulation. Low-current micro- 
stimulation (biphasic pulses at 100 Hz; 100-ms pulse width; 75 pA current) 
applied to M1 evoked clear motor responses and EMG bursts (36), whereas 
such low currents did not evoke motor responses when applied to S1. 

Overall, the anatomical landmarks found during the implantation surgery, 
examination of RFs, and microstimulation tests assured us that M1 and $1 
were localized correctly. 


Primate Setup. We used a custom modified primate chair (Crist Instruments). 
The chair had armrests that were angled at 23° from the horizontal plane to 
position the monkey arms parallel to the screen. Monkeys’ arms were placed 
on the armrests and restrained with soft braces. Special care was taken to 
remove postural incongruency that could weaken and eliminate the RHI. We 
started with pilot studies to find the right way to position the arm rests, the 
chair, and the monitor so they did not permit a direct view of the arms but 
allowed an appropriate view of the virtual arms. We scaled the arms in the 
visualization software to approximate the size of the physical arms. The LCD 
screen was positioned at 40 cm away from the monkeys’ eyes and raised by 
just 10 cm relative to the armrest. 


Behavioral Training. Monkeys were introduced to the experimental apparatus 
several weeks before the recording sessions. Calm sitting in the chair with 
restrained arms while gazing at the screen was reinforced with fruit juice. The 
brushing stimuli were then introduced. Although both monkeys initially 
reacted with movements upon being brushed, gradually they acclimated to 
the stimuli and relaxed. 


EMG Recording. Surface EMGs were recorded with a 16-channel EMG system 
(Delsys). We recorded from the following muscles on both arms: biceps, 
deltoid, triceps, wrist flexors, and wrist extensors. 


Robotic Brush. To assure the spatial congruency of the visual and physical 
stimuli, we used a custom 2D serial Cartesian robot with fast control-loop 
(1 kHz) and precise (i.e., submillimeter) positioning. A soft brush was mounted 
on the end effector of the robot. A position controlled Maxon DC motor, 
equipped with encoders, precisely positioned the robotic actuator to the 
trajectory that matched the brush location shown on the screen. This control 
strategy resulted in accurate, reliable, and repeatable touches of the monkey 
arm. The skin locations touched by the physical brush were in good corre- 
spondence with the locations touched by the virtual ball. This was assured 
through the following steps. The monkey's arm was shaved before the ex- 
perimental sessions so we could precisely designate the location and onset 
of the physical touch and match the spatial and temporal characteristics of 
the physical brushing with the virtual touch. To set the touch points, the 
robot end-effector was brought in close contact with the monkey arm. 
The motors were turned off, but the encoders were active and recorded the 
precise positions of each touch point. The corresponding touch points for 
the virtual ball were programmed in the virtual reality software (Autodesk 
MotionBuilder). The instances of the brush making a contact with the skin 
were detected offline using the positions registered by the encoders of the 
robot motors. Fig. $4 shows the robot's path recorded by the encoders. 


Eye Tracking and Rewards. The monkeys’ heads were unrestrained, and their 
gaze direction was monitored continuously. Custom eye tracking was 
designed for the purpose of the experiment. The tracking was based on the 
open-source TLD/Predator tracking algorithm (37). The monkey was ran- 
domly rewarded for looking at the screen every 350 + 50 ms. The reward 
duration started at 35 ms and increased to 70 ms by the end of the session to 
maintain motivation. Periods when the monkey looked away or closed its 
eyes were not rewarded. Trials for which monkeys looked at the screen less 
than 90% of the trial duration were dismissed from the analyses. 
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Neural Responses Analysis. To evaluate response magnitude, a 200-ms win- 
dow was centered at the peak response estimated from the PSTH. Response 
rate was calculated as the average firing rate within this window. Responses 
amplitude, AR, was calculated as the difference between this response rate 
and the mean baseline rate before the stimulus (from —1.2 to —0.2 s relative 
to stimulus onset). We used raw and normalized rates to calculate AR. 
Normalized neuronal rates (z-scores) were derived from the raw rates by 
subtracting the mean and dividing by SD. Normalized rates were useful for 
both population plots and population averages because they removed a bias 
in favor of neurons with high firing rates. 


Monkey Avatar. The monkey avatar was designed with Autodesk 3Dsmax 
software based on the morphology of monkey M. Custom texture was 
designed based on the mixed real texture of monkey M and monkey N. 
Monkey avatar arms were shown from a first-person perspective on a uniform 
gray background. The virtual ball was represented by a white sphere. 
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Cross-Correlation Analysis of Response Latency. Latencies of Vonly and V+P 
sync responses were compared using a cross-correlation analysis of the cor- 
responding PSTHs. The lag between these responses was determined as the 
time of the peak in cross-correlation function. We preferred this method 
rather than calculating latencies from the PSTHs to avoid an overestimation 
of latency for weaker Vonly responses. Cross-correlation functions were 
calculated for all possible pairs of neurons. Distributions of peak times these 
functions (i.e., lags between Vonly and V+P sync) are shown in Fig. 3 E and J. 
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BRAIN-MACHINE INTERFACES 


A Brain-Machine Interface Enables Bimanual Arm 
Movements in Monkeys 
Peter J. Ifft,""? Solaiman Shokur,2? Zheng Li,” Mikhail A. Lebedev,”* Miguel A. L. Nicolelis’?>** 


Brain-machine interfaces (BMls) are artificial systems that aim to restore sensation and movement to paralyzed 
patients. So far, BMls have enabled only one arm to be moved at a time. Control of bimanual arm movements 
remains a major challenge. We have developed and tested a bimanual BMI that enables rhesus monkeys to control 
two avatar arms simultaneously. The bimanual BMI was based on the extracellular activity of 374 to 497 neurons re- 
corded from several frontal and parietal cortical areas of both cerebral hemispheres. Cortical activity was transformed 
into movements of the two arms with a decoding algorithm called a fifth-order unscented Kalman filter (UKF). The 
UKF was trained either during a manual task performed with two joysticks or by having the monkeys passively ob- 
serve the movements of avatar arms. Most cortical neurons changed their modulation patterns when both arms were 
engaged simultaneously. Representing the two arms jointly in a single UKF decoder resulted in improved decoding 
performance compared with using separate decoders for each arm. As the animals’ performance in bimanual BMI 
control improved over time, we observed widespread plasticity in frontal and parietal cortical areas. Neuronal rep- 
resentation of the avatar and reach targets was enhanced with learning, whereas pairwise correlations between 
neurons initially increased and then decreased. These results suggest that cortical networks may assimilate the 
two avatar arms through BMI control. These findings should help in the design of more sophisticated BMIs capable 


of enabling bimanual motor control in human patients. 


INTRODUCTION 


The complexity and variety of motor behaviors in humans and other 
primates is greatly augmented by the remarkable ability of the central 
nervous system to control bimanual movements (1). Yet, this func- 
tionality has not been enacted by previous brain-machine interfaces 
(BMIs). BMIs are hybrid systems that directly connect brain tissue 
to machines to restore motor and sensory functions to paralyzed in- 
dividuals (2). The advancement of BMIs has been driven by two fun- 
damental goals: investigation of the physiological principles that guide 
the operation of large neural ensembles (3) and the development of 
neuroprosthetic devices that could restore limb movements and sen- 
sation to paralyzed patients (4, 5). Previous BMIs mimicked only single- 
arm control and were represented by either a computer cursor (6-11), 
a robot (8, 12, 13), or an avatar arm (14), but did not enable simulta- 
neous bimanual control of arm movements. 

Studies in nonhuman primates have shown that the brain does not 
encode bimanual movements simply by superimposing two indepen- 
dent single-limb representations (15, 16). Cortical regions, such as the 
supplementary motor area (SMA) (17-19) and the primary motor 
cortex (M1) (16, 18, 19), exhibit specific patterns of activity during bi- 
manual movements. These complex neuronal representations pose a 
major challenge for BMIs enabling bimanual control because they 
cannot be designed simply by combining single-limb modules. Here, 
we tested the ability of a bimanual BMI to enable rhesus monkeys to 
control two avatar arms simultaneously. 
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RESULTS 


Large-scale recordings and experimental paradigms 

We set out to discover whether large-scale cortical recordings could 
provide sufficient neuronal signals to accurately control a bimanual 
BMI (4, 20). We implanted volumetric multielectrode arrays in two 
monkeys (768 microelectrodes in monkey C; 384 microelectrodes in 
monkey M) (Fig. 1A) as described previously (20). Neural signals were 
sorted using template matching algorithms within commercially avail- 
able software (Plexon Inc.). In monkey C, we simultaneously sampled 
(Fig. 1, C, E, and F) from the SMA (73 to 110 units in the left hemisphere, 
0 to 20 units in the right hemisphere; ranges for all experiments), M1 
(176 to 218 units in the left hemisphere, 45 to 62 units in the right hem- 
isphere), primary somatosensory cortex (S1) (9 to 64 units in the left 
hemisphere, 0 to 34 units in the right hemisphere), and posterior 
parietal cortex (PPC) (0 to 4 units in the left hemisphere, 22 to 47 units 
in the right hemisphere). In monkey M, we sampled from M1 (80 to 
90 units in the left hemisphere, 195 to 204 units in the right hemisphere) 
and S1 (47 to 56 units in the left hemisphere, 127 to 149 units in the right 
hemisphere). The daily unit count neared 500 for each monkey, which 
constitutes the highest number of simultaneously recorded units in 
nonhuman primates to date (21). The high unit count for monkey M 
persisted for 48 months after the implantation surgery, and for monkey 
C persisted for at least 18 months after the surgery (recordings are still 
continuing in these two animals). 

Using this large-scale BMI, both monkeys were able to directly con- 
trol the simultaneous reaching movements performed by two avatar 
arms. Moreover, these monkeys learned to operate the bimanual BMI 
without producing overt movements of their own arms. A realistic, vir- 
tual monkey avatar (fig. S1A) was chosen as the actuator for the bi- 
manual BMI because in previous studies (14, 22) and experiments for 
this study (Fig. 2), we observed that monkeys readily engaged in both 
active control and passive observation of the avatar movements. Each 
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monkey observed two avatar arms on a computer monitor from a first- 
person perspective (Fig. 1B). A trial began with the appearance on the 
screen of two square targets. Their position was the same in all trials, 
and they served as the start positions for the avatar hands. The mon- 
key had to place the avatar hands over their respective targets and hold 
these positions for a delay, randomly drawn from a uniform distribu- 
tion (400- to 1000-ms intervals, Fig. 1D). The two squares were then 
replaced by two circular targets in 1 of 16 possible configurations (right, 
left, up, or down relative to start position for each hand). At this point, 
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the monkey had to place both avatar hands over the targets indicated by 
the two circles and hold the targets simultaneously for a minimum of 
100 ms to receive a fruit juice reward. In the unimanual version of this 
task, a single avatar arm had to reach for a single target. 

The tasks were performed in three possible ways: joystick con- 
trol, brain control with arm movements (BC with arms), and brain 
control without arm movements (BC without arms). Both monkeys 
learned to perform BC without arms, but through different learn- 
ing sequences. Monkey C began with joystick control, during which 
the right and left avatar arms were con- 
trolled directly by movements of the two 
joysticks (Fig. 1F) (18). Monkey C then 
learned BC with arms, during which move- 
ments of the avatar arms were controlled 
directly by cortical activity, although the 
monkey was permitted to continue ma- 
nipulating the joysticks. Finally, monkey 
C learned BC without arms, a mode of 
operation where decoded brain activity 
once again controlled avatar arm move- 
ments, but now overt limb movements 
were prevented by gently restraining both 
arms. Monkey M did not use the joystick 
in any task. Rather, this monkey’s task 
training began by having it passively ob- 
serve the avatar arms moving on the screen 
as an initial step before learning BC with- 
out arms. This type of BMI training has 
clinical relevance for paralyzed subjects 
who cannot produce any overt movements, 
and it has been used in several human 
studies (13, 23). 

To set up BC with arms for monkey 
C, we followed our previously established 
routine (8, 10) of training the BMI de- 
coder on joystick control data to extract 
arm kinematics from cortical activity. 
Daily sessions dedicated solely to joystick 
control lasted 20 to 40 min. Brain control 
sessions began with 5 to 7 min of the joy- 
stick control task, before switching to BC 
with arms for the final 20 to 40 min. De- 
spite more complexities regarding inde- 
pendent control of two virtual limbs, the 
decoding accuracy for our bimanual BMI 
was sufficient for online control (movie 
$3) and matched the accuracy previously 
reported for less challenging unimanual 
BMIs (7, 8, 10, 24, 25). 
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Fig. 1. Large-scale electrode implants and behavioral tasks. (A) Monkey C (left) and monkey M 


(right) were chronically implanted with eight and four 96-channel arrays, respectively. (B) The monkey 
is seated in front of a screen showing two virtual arms and uses either joystick movements or mod- 
ulations in neural activity to control the avatar arms. (C) Four hundred forty-one sample waveforms 
from typical monkey C recording sessions, with the color of the waveform indicating the recording 
site [shown in (A)]. (D) Left to right: Trial sequence began with both hands holding a center target for 
a random interval. Next, two peripheral targets appeared, which had to be reached to and held with 
the respective hands to receive a juice reward. (E and F) Raster plot of spike events from 438 neurons 
(y axis) over time (x axis) for a single unimanual (E) and bimanual (F) trial. Target location and position 


traces of trial are indicated to the right of the raster panel. 
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Bimanual joystick control 

Monkey C was trained to perform both 
unimanual and bimanual joystick control 
tasks very accurately (greater than 97% of 
the trials were correct) (fig. S1, B to E, and 
movies S1 and 2). Cortical recordings 
collected from this monkey revealed wide- 
spread neuronal modulations that reflected 
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movement timing and direction (Figs. 1, E and F, and 3 to 5). Con- 
sistent with previous studies (15, 16, 18), cortical activity from multiple 
areas was different between unimanual and bimanual movements (Figs. 
3 and 4). In motor areas M1 (Fig. 3, A, B, E, and F) and SMA (Fig. 3, C, 
D, G, and H), individual units (Fig. 3, A, C, E, and G) and neuronal 
populations (Fig. 3, B, D, F, and H) alike exhibited directionally selec- 
tive modulations during both unimanual and bimanual performance. 
For each configuration of the pair of targets, we characterized neuro- 
nal modulations as Az—the difference between the movement epoch 
(from 150 to 600 ms after target appearance) firing rate and baseline 
rate, both expressed in normalized units (z scores). Normalization to z 
scores was applied to each unit’s firing rate before any grouping or 
averaging of individual trials. Average modulation for all target posi- 
tions was quantified as absolute value of Az averaged for all target po- 
sitions (|Az|). Directional selectivity was measured as the SD of Az for 
different target positions, o(Az). 

The transition from unimanual to bimanual movements (table $1) 
induced several effects in M1 and SMA. First, we observed a promi- 
nent increase in |Az| during bimanual movements by 76.7 and 34.6% 
for left M1 and right M1, respectively, and 35.8 and 37.9% for left and 
right SMA (P < 0.01, t¢ test). M1 neurons exhibited clear preference for 
the contralateral rather than the ipsilateral arm during unimanual per- 
formance, both in terms of overall modulations (28.3% increase in |Az| 
for contralateral versus ipsilateral arm; P < 0.01) and in terms of tuning 
depth [22.3% increase in o(Az)]. An opposite, ipsilateral preference was 
observed for SMA [19.1% decrease in |Az| and 11.1% decrease in o(Az); 
P < 0.01]. For both M1 and SMA, directional tuning depth during the 
bimanual task was about equal for the left and right arm [left o(Az): 0.08; 
right o(Az): 0.09; P > 0.01]. Notably, SMA was the only area where more 
neurons were tuned to both arms after a transition from unimanual to 
bimanual movements (P < 0.01) (Fig. 4A). In addition to changes in 
overall modulations and directional tuning depth, bimanual control 
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Fig. 2. Comparison of bimanual behavioral training with cursor and 
avatar actuators. (A and B) Two 2D cursors (A) or two avatar arms (B) were 
controlled by joystick movements. In both environments, the target for 
each hand was a white circle. (C and D) Percentage of total trials containing 
a threshold amount of movements (avatar arm reached beyond 80% of 
distance from center to target), with the left arm (C) or right arm (D) shown 
in lower panels. The first 10 sessions of avatar and cursor bimanual training, 
conducted on alternate days, are shown separately in blue and red. *P < 
0.05, Student's t test. 
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Fig. 3. Modulation of cortical neurons during manually performed 
unimanual and bimanual movements. (A) Representative left M1 neu- 
ron peri-event time histogram (PETH) aligned on target appearance 
(gray line) for each of 16 left and right target location combinations dur- 
ing bimanual movements. Below the 4 x 4 grid are corresponding 
PETHs for the same neuron during unimanual trials in each of the four 
directions. (B) Same layout as (A) for the population of left M1 neurons. 
Each row of each color plot panel represents a single neuron, and the 
pixel color is the normalized firing rate or z score (color scale at bot- 
tom). (C and D) Representative neuron (C) and neuronal population (D) 
in the SMA of the brain. (E to H) Az for each of the four movement 
directions for unimanual (red) and bimanual (blue) trials for the left (top) 
and right (bottom) arms: for one M1 neuron (E), for a population of M1 
neurons (F), for one SMA neuron (G), and for a population of SMA neu- 
rons (H). 
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Fig. 4. Directional tuning during bimanual versus unimanual move- 
ments. (A) Fraction of neurons in each cortical area that had significant 
tuning to both arms during unimanual (red) and bimanual (blue) trials, 
determined from regression. (B) Absolute value of the difference between 
preferred direction of the contralateral arm computed from bimanual and 
unimanual trials. Data are shown separately for each cortical area. (C) Same 
analysis as (B) but showing the difference in preferred direction for the ip- 
silateral arm. All data are means + SEM. Analysis was compiled from activity 
of 492 M1 neurons, 203 SMA neurons, 90 S1 neurons, and 61 PPC neurons. 


resulted in changes in neuronal preferred directions, which shifted be- 
tween the unimanual and bimanual conditions by 53.1 + 4.0° (mean + 
SEM) for the contralateral arm and 66.0 + 5.4° for the ipsilateral arm 
(Fig. 4, B and C). 


Offline decoding of bimanual movements with 

unscented Kalman filter 

The unscented Kalman filter (UKF) (24) was selected for the BMI de- 
coder in this study. The UKF is a decoding algorithm that has been 
used previously to extract motor commands from brain activity to 
control a cursor (24) or a virtual arm (14) in real time. The UKF repre- 
sents both the reaching parameters, such as position and velocity (state 
model), and their nonlinear relationship to neuronal rates (tuning mod- 
el). Additionally, it incorporates a history of neuronal rates (the higher 
the UKF order, the richer the history). Here, the UKF was updated 
every 100 ms based on the previous state and the neuronal activity 
recordings. The UKF was first applied to cortical recordings obtained 
during the joystick control task in monkey C. To decode the position 
of one arm during unimanual movements, we used a UKF with a two- 
dimensional (2D) output (X and Y coordinates of the hand; Fig. 5, A 
and E). To decode bimanual movements, we applied either two 
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separate 2D UKFs (Fig. 5B) or a UKF with a 4D output (X and Y 
for both hands; Fig. 5, D and F). 

The complexity of neuronal representations of bimanual movements 
became apparent from the neuron-dropping curves (8, 10). Neuron- 
dropping curves describe the deterioration of decoding accuracy as neu- 
rons are removed (dropped) from the population used for decoding. 
They are a useful analytical method for showing the effect of neural en- 
semble size on the ability to decode motor characteristics, measured as 
correlation coefficient r. Here, this method clearly indicated that more 
neurons were needed to achieve the same decoding accuracy for each 
arm during the bimanual task than during the unimanual task (com- 
pare Fig. 5A with Fig. 5B). For example, to achieve decoding accuracy of 
r = 0.5 with the 2D UKF, a mean of 80 neurons (drawn from the full 
population) were needed for unimanual hand control and 145 neurons 
for bimanual hand control, despite using the same 2D UKF for each 
hand. The decoding accuracy, quantified as r, was proportional to the 
logarithm of neuron count in each case (Fig. 5, A to D). Additionally, 
bimanual movements required a longer time to train the UKF than uni- 
manual movements (fig. S2A). Furthermore, individual neurons more 
strongly contributed to the decoding of one of the arms when move- 
ments were unimanual, but more homogeneously represented each 
arm during bimanual movements (fig. $2, B and C). Both distribu- 
tions were leftward shifted from the null distribution (fig. S2D) col- 
lected from the same recording session but during periods without 
task execution (P < 0.05, both right and left arms, Wilcoxon signed 
rank test). We obtained better bimanual prediction accuracy when 
the 2D UKF was trained on bimanual movements compared to the 
same model trained on unimanual movements of each arm separately 
(fig. S2E, P < 0.01). Similarly, training the UKF on bimanual move- 
ments yielded more accurate predictions for bimanual than for uni- 
manual movements (fig. S2F, P < 0.01). 

During bimanual joystick control, the position of the right and 
left arms was decoded from multi-area ensemble activity with high 
accuracy (r = 0.85 + 0.02 and r = 0.62 + 0.03, fig. S4C). Looking at 
cortical areas separately, the best decoding was achieved with M1 
neurons (n = 245; r = 0.73 + 0.03, average of two-arm r values). A 
less numerous population of SMA neurons performed worse (n = 73; 
r = 0.22 + 0.02), but the contribution from SMA and other areas to the 
overall predictions was still evident from the rise of the entire ensem- 
ble dropping curve beyond the maximum M1 performance, as well as 
the steady rise of the individual area dropping curves (Fig. 5, A, B, and 
D). Moreover, when UKF predictions were run for individual neurons 
and all neurons were ranked by the accuracy of these predictions, many 
non-M1 neurons received high rankings. For example, of the 50 top- 
ranked neurons, 27 (54%) were from M1, 16 (32%) from SMA, 4 (8%) 
from PPC, and 3 (6%) from S1. Therefore, although M1 neurons con- 
tributed the greatest amount to kinematic predictions, non-M1 areas 
such as SMA, PPC, and S1 provided informative signals as well. 

In addition to predictions of the coordinates of two hands, the dis- 
tance between hands was represented with similar accuracy (Fig. 5C) 
when derived from the predictions of two hand positions made with 
the 4D UKF model. 


Brain control of bimanual movements with UKF 

After testing the 4D UKF on monkey C’s joystick control data and 
finding that it consistently outperformed the 2D UKF on the joystick 
control data (Fig. 5D and fig. S3A), we chose this decoder for real-time 
BMI control. This decoder incorporated kinematics of both arms in 
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Fig. 5. Neuron-dropping curves for joystick control. (A to D) Neuron-dropping curves for (A) uni- 
manual joystick control, (B) bimanual joystick control using two 2D decoding models, (C) interhand 
spacing, and (D) bimanual joystick control using one 4D decoding model. Curves are shown separately 
for each area, indicated by color. (E and F) Offline predictions using 2D UKF for unimanual movements 


(E) and 4D UKF for bimanual movements (F). 


the state variables, thus allowing the algorithm to jointly represent 
both arms in a single model. Even with this improved decoder, switch- 
ing from hand control (fig. SIE) to BC with arms resulted in a sharp 
drop in monkey C’s performance level during initial sessions (Fig. 6C). 
However, over the next 24 sessions of BC with arms, monkey C’s pro- 
ficiency improved substantially in both bimanual task performance 
(that is, percent of trials where both arms reached their targets with- 
in the maximal allowed trial duration; Fig. 6C) and individual arm 
performance (that is, percent of trials where a single arm reached its 
correct target within the same duration; Fig. 6D). Additional improve- 
ment in performance was achieved after the decoder was upgraded 
from a first-order UKF to a fifth-order UKF (Fig. 6C and fig. S3, B 
to D). Changing from a first to a fifth order provided a more detailed 
history of previous neuronal activity to be incorporated into the decoder. 
By the end of BC with arms training, a consistent level of performance 
exceeded 70% correct trials (Fig. 6C), including more than 90% correct 
trials for each arm individually (Fig. 6D). 
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Our BC without arms experiments 
were conducted in both monkeys. They 
were designed to match the practical needs 
of paralyzed people who have to learn 
BMI control without being able to produce 
overt upper limb movements. Previous- 
ly, to reach the same goal, single-effector 
BMI studies have used a coadaptive de- 
coding model with iteratively updated 
tuning properties (25), requested subjects 
to imagine movements (13, 26) or had 
them passively observe effector movements 
(27, 28). In our study, a passive observation 
paradigm became the basis for how the 
BMI decoder was trained without requir- 
ing the monkeys to produce overt arm 
movements. For these experiments, we 
had the monkey passively observe the 
movements of the avatar while its arms 
were gently restrained (Fig. 6A). The screen 
displayed the two avatar arms moving in 
center-out trajectories toward the targets. 
The trajectories were either replayed from 
a previous joystick control session (monkey 
C) or preprogrammed using estimates of 
kinematic parameters (monkey M). These 
bimanual passive observation movements 
were clearly reflected by cortical modu- 
lations (fig. S4, A to C). Neuronal mod- 
ulations to passive observations do not 
appear to be explained by muscle activity 
(fig. S5, A and B). To explore these neu- 
ronal modulations, we applied the 4D 
UKF to extract passive observation move- 
ments of the avatar arms from cortical ac- 
tivity (Fig. 6B). The accuracy of these 
extractions was different for the left and 
right avatar arms (monkey C: r = 0.46 + 
0.05 for the left avatar arm versus r = 
0.12 + 0.05 for the right; monkey M: r = 
0.47 + 0.03 versus r = 0.23 + 0.02) (fig. 
S4C) and fluctuated in time (r in the range of —0.29 to 0.64; fig. S6). 
The running accuracies for the two arms were weakly positively cor- 
related between themselves (correlation coefficient of 0.25 + 0.12 in mon- 
key C and 0.31 + 0.14 in monkey M, across all sessions; mean + SEM), 
likely reflecting drifts in the level of overall attention to both arms. One 
could speculate in addition that attention was occasionally unevenly dis- 
tributed between the two avatar arms (for example, negative correlation 
of running accuracies during the interval 155 to 200 s in fig. $6), but this 
issue will have to be more carefully investigated in future studies using 
more precise eye-tracking methods. 

After the UKF was trained on 5 to 7 min of passive observation, the 
mode of operation was switched to BC without arms for 25 to 45 min 
(movie $4). The monkeys controlled the avatar with their cortical activ- 
ity while still fully arm-restrained. Both monkeys rapidly improved the 
performance for each arm in the bimanual BC without arms task within 
5 to 10 sessions (monkey C rising from 43 to 79%; monkey M from 
15 to 62%; Fig. 6C, black circles). Similar to the passive observation 
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Fig. 6. Passive observation and brain control paradigms. (A) A mon- 
key was seated in front of a screen with both arms gently restrained 
and covered by an opaque material during passive observation and BC 
without arms experiments. (B) Actual left and right arm X position 
(black) compared with predicted X position (red) for passive observa- 
tion sessions. Pearson's correlation, r, is indicated. (C) Performance of 
monkey C (left) and monkey M (right) quantified as fraction correct 
trials. Shown separately for monkey C are different decoding model 
parameter settings (red, blue markers) as well as brain control without 
arm movement (BC without arms) sessions (black, both monkeys). Ses- 
sions with less than 10 attempted trials were set to zero because of in- 
sufficient data (open circles). (D) Fraction of trials where the left arm 
(green circles) and right arm (blue circles) acquired their respective 
target during brain control. Linear fit for learning trends of each par- 
adigm is shown in (C) and (D). (E and F) Fraction of correct predic- 
tions by k-NN of target location for each arm (blue/green) over the 
trial period during both passive observation (E) and BC without arms 
(F) in both monkey C (left column) and monkey M (right column). (G) 
Mean k-NN target prediction fraction correct from neuron-dropping 
curves separated by cortical area for each monkey [same columns as 
(E) and (F)]. 
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pattern (Fig. 6, E and F, fig. S4C), the performance accuracy for the 
left and right arms was unequal during the BC without arms task 
(monkey C: left, 98.5%; right, 94.4%; monkey M: left, 96.4%; right, 
77.7%; average over the last three BC without arms sessions). The 
first four BC without arms sessions for monkey M (open circles, Fig. 
6C) contained fewer than 10 trials which met the minimum attention 
threshold and were counted as zero accuracy. To compute a chance 
performance level for each mode of control, we performed a shuffled- 
target analysis (Fig. 6C). Cursor trajectories of each trial were replayed 
with peripheral target locations drawn from a randomly shuffled set of 
target combinations. Correct performance was defined the same way 
as in online sessions, where both center and peripheral targets must 
be acquired within the 10-s timeout limit. Chance-level performance 
derived from the shuffle test was very low for BC with arms data (less 
than 10% correct trials, Fig. 6C) and slightly higher but still far below 
monkey performance for BC without arms data (20 to 30% for mon- 
key C, 10 to 20% for monkey M; Fig. 6C). For 20 of the 21 BC without 
arms sessions, monkey performance was statistically greater than 
chance performance (P < 0.05, ¢ test), the lone exception being the 
second session for monkey M. 

To obtain further evidence that cortical modulations during brain 
control sessions reflected the goal of the bimanual task, we evaluated 
how cortical ensembles represented the location of targets on each 
trial. It was essential that we decoded target position instead of posi- 
tion or velocity of the avatar arms in this analysis. Unlike the position 
of the avatar arms, target locations were not algorithmically linked to 
neuronal activity during real-time BMI control. Therefore, this analy- 
sis was not confounded by the fact that neuronal tuning to arm kin- 
ematics was preset by the UKF tuning model. Such a confound would 
have been a hindrance if we attempted to derive tuning from the re- 
lationship between movement direction and neuronal rates. In our ex- 
periments, the target locations were neither a parameter of the UKF 
nor its output; thus, when we examined how those locations were 
tracked by cortical activity, we were certain that we dealt with a true 
representation of movement goals and not with an epiphenomenon 
of the real-time BMI. Note that the UKF model may have influenced 
this analysis indirectly because it drove the avatar arms, and the position 
of the arms likely affected the neuronal representation of the targets. 
We used a k-nearest neighbor (k-NN) classifier to extract target loca- 
tions from cortical modulations (Fig. 6, E to G). To quantify cortical re- 
presentation of each target, which potentially could be different for 
neurons from different hemispheres and/or areas, we used two clas- 
sifiers, one for each target. Both target locations were clearly reflected 
by cortical ensembles, starting with the appearance of the targets and 
continuing throughout the trial (Fig. 6, E to G). Cortical activity re- 
presented the targets during both BC without arms sessions (Fig. 6, F 
and G) and passive observation sessions (Fig. 6E). The accuracy of 
each representation was measured as fraction correct. Using k-NN, 
we decoded the target location of the left arm more effectively than 
the right arm in both monkeys (Fig. 6F). This matches the behav- 
ioral results from Fig. 6D, which showed better BC without arms 
performance with the left hand of both monkeys as well. Despite 
this difference, both right and left target locations could be decoded 
at significant levels within the same epoch after target appearance. 
This dual representation persisted through the reaction time and 
movement epoch of a typical trial (Fig. 6, E and F). 

We next assessed the effect of the number of recorded neurons 
and relative contributions of cortical areas on k-NN decoding of target 
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position. Similar to Fig. 6F, k-NN decoded left and right target loca- 
tions during the BC without arms task. The mean prediction accuracy 
for both arms improved approximately linearly with the logarithm of 
neuronal ensemble size (for example, for monkey M, M1 ensembles: 
fraction correct = 0.34 for n = 5, fraction correct = 0.37 for n = 10, and 
fraction correct = 0.43 for n = 100, with a chance level of 0.25; Fig. 6G, 
right). This followed the same trend observed for prediction of arm 
kinematics (Fig. 5, A to D). For monkey M, with microelectrodes im- 
planted in both leg and arm M1 areas, the targets were better repre- 
sented by neurons located in the arm area (fraction correct: 0.43 + 
0.02, n = 100) compared to the neurons in the leg area (fraction cor- 
rect: 0.31 + 0.03, n = 100; P < 0.01). Neurons in the leg area of M1 did, 
however, contribute to predicting target location: a population of 
about 100 neurons in the leg area of M1 rivaled in accuracy 100 neu- 
rons located in the S1 arm region. 


Neuronal plasticity during BC without arms training 

In parallel with each monkey’s learning of the BC without arms task, 
we observed plastic changes in the firing patterns of cortical ensem- 
bles. These changes were particularly clear in the functional reorgani- 
zation of the cortical representation of the avatar during the passive 
observation task, which was measured at the beginning of each session 
of BC without arms (Fig. 7, A and B). The decoding accuracy of pas- 
sively observed avatar kinematics (measured as prediction r) was clearly 
enhanced as the training progressed. 

Furthermore, we observed a gradual reduction [P < 0.01, anal- 
ysis of variance (ANOVA)] in firing rate correlations among cortical 
neurons as animals were training in the BC without arms task (Fig. 7, 
B to E). During early BC without arms sessions, correlations between 
neurons were 1.7 to 2.2 times greater than during passive observation 
periods tested on the same day. Over the next few days, however, these 
cortical correlations decreased until they reached the same level as 
during passive observation (Fig. 7B). During this reduction, correla- 
tions between neurons from the same hemisphere (solid red line, 
Fig. 7C) and the same cortical area (solid blue line) remained higher 
than the correlations between neurons from different hemispheres or 
areas (dashed lines). An ANOVA test showed that both area and 
hemispheric relationships were factors related to the decrease in cor- 
relation (P < 0.01). In both monkeys, there was a greater proportional 
decrease in cross-hemisphere correlation during BC without arms 
learning than within-hemisphere correlation (monkey C: 85.2% reduc- 
tion across hemispheres versus 54.0% within hemisphere; monkey M: 
56.6% across hemispheres versus 36.1% within hemisphere). Similarly, 
correlations between cortical areas (same hemisphere) decreased more 
than those within an area (monkey C: 76.6% reduction between cortical 
areas versus 54.0% within cortical area; monkey M: 53.7% reduction 
between cortical areas versus 29.9% within cortical area). 


DISCUSSION 


Our findings suggest that BMI technology can be applied to the chal- 
lenging task of enabling bimanual control in subjects who do not 
produce overt arm movements. The current study enhances the degree 
of sophistication of an upper limb BMI by enabling simultaneous control of 
two arms. This was achieved by introducing a bimanual decoding/training 
paradigm—one that takes full advantage of large-scale simultaneous 
bihemispheric chronic cortical recordings of up to 500 neurons, a virtual 
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Fig. 7. Cortical plasticity during passive observation and brain con- 
trol without arm movements. (A) UKF prediction performance r over 
time using passive observation data from the beginning of each session 
to predict observed movement trajectories. (B) Mean correlation r of 
neural firing among recorded neuronal populations throughout passive 
observation and brain control without arm movement (BC without arms) 
training sessions. (C) Mean inter- and intrahemispheric (red) and inter- 
and intra-area (blue) correlation 7 versus session. (D) Neuron versus neu- 
ron correlation indicated by pixel color for two monkeys on the first (left) 
and last (right) day of BC without arms training. Within each panel, neu- 
rons are sorted by cortical area and mean correlation strength. (E) Same 
as (D), except for monkey M. (A to C) Left column, monkey C; right column, 
monkey M. 
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reality training environment, an optimal bimanual decoding 
algorithm, and the recently discovered (22) phenomenon of visually 
driven cross-modal cortical plasticity. 

From our earlier studies, we have argued that BMIs provide impor- 
tant insights into the physiological principles that govern the function 
of brain circuits (3). In this context, the present study tested whether 
these principles apply to bimanual motor control. To examine the role 
of neural ensembles for bimanual control, we used simultaneous, very 
large scale recordings from fronto-parietal cortical ensembles (2, 3) 
and obtained extractions of bimanual motor parameters. Neuron- 
dropping curves constructed from both manual control data (Fig. 5, 
A to D) and real-time BC without arms data (Fig. 6G) indicate that 
the accuracy of neural decoding steadily and linearly improves with 
the logarithm of neuronal ensemble size. With our distributed mul- 
tisite recording approach, we demonstrated the representation of bi- 
manual movements of several fronto-parietal cortical areas. This result 
is consistent with previous neurophysiological studies of bimanual 
motor control (15-19). From our data, the M1 contribution to the 
decoding was the most substantial, which is likely due to a higher 
proportion of task-related neurons. Supplementing the M1 ensem- 
ble (red line in Fig. 5, A to D) with neuronal data from the other 
areas (black line) further improved the decoding. Furthermore, 
non-M1 areas such as PPC (29) and dorsal premotor cortex (PMd) 
(8, 10, 30) are reliable sources for BMI decoding with an overlapping 
but often distinct role from M1 output (31-33). This is especially true 
in an area such as SMA, which is involved in bimanual coordination 
(17, 19). 

Our results support a highly distributed representation of bimanual 
movements by cortical ensembles, with individual neurons and neural 
populations representing both avatar arms simultaneously (17-19). 
Most neurons recorded in this study contributed to the predictions 
of kinematics of each arm (fig. $2, B to D). Moreover, during online 
BMI control of bimanual movements, such multiplexing of the kine- 
matics of both arms by individual neurons became even more prom- 
inent (fig. S2C). 

It should be emphasized that the distributed cortical representation 
of bimanual movements could not be described as a linear superpo- 
sition of the representations of unimanual movements. Most clear in 
SMA and M1 brain regions (Fig. 3), but also evident in $1 and PPC 
(Fig. 4), the activity of individual neurons and neural populations dur- 
ing bimanual tasks was not a weighted sum of unimanual patterns 
derived from data collected from right and left arms independently. 
These observations point to an existence of a separate, bimanual state 
of the network in which modulations represent the movements of 
both arms simultaneously by way of nonlinear transformations of 
the separate neural tuning profiles of each arm (Fig. 3, A to D). At 
this point, we can only speculate about the function of this nonlinearity. 
From one moment to the next, the two arms need to be able to switch 
between unimanual and bimanual functionality. During unimanual 
control, it is important that the motor drive to the nonworking arm 
is inhibited. Conversely, during two-arm behaviors, it becomes impor- 
tant that a motor program in one arm does not interfere with the other 
arm and does not evoke unwanted synergies in both arms, but rather 
permits a degree of independence. Nonlinear phenomena have been re- 
ported at the behavioral level in bimanual motor control studies, which 
showed that motor systems can choose to favor stability (interlimb co- 
ordination, nonlinear) over more unstable (interlimb independence, 
linear) (34-36) motor programs in a task-dependent way. 
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Overall, our findings provide support for the notion that very large 
neuronal ensembles, not single neurons, may define the true physiolog- 
ical module underlying normal motor functionality (2, 3, 10, 14, 37). 
This suggests that small cortical neuronal samples may be insufficiently 
informative to control complex motor behaviors using a BMI (9, 25, 38, 39), 
but BMIs with few neurons could be useful test beds for experimenta- 
tion with less challenging motor tasks. Although we did not attempt 
bimanual control with just a few neurons in this study, we probed small 
ensembles indirectly by using a neuron-dropping analysis using a kK-NN 
classifier to estimate the contribution of ensemble size to target repre- 
sentation during BC without arms (Fig. 6, E to G). This analysis showed 
that large ensembles always performed better. 

Our study provides new insights into the plasticity of cortical en- 
sembles. Previously, we demonstrated that cortical ensembles can 
undergo substantial plasticity during learning of BMI tasks (10). We 
even observed an emergence of visually evoked responses in $1 and 
M1 when attended to virtual avatar arms (14, 22). Here, we observed 
improvements in performance as the monkeys enacted real-time BMI 
control of bimanual movements. These improvements were accompa- 
nied by a steady decline of correlated neural activity throughout the 
four recorded cortical areas and in both hemispheres (Fig. 7, B to E). 
Previous work (3, 40, 41) has identified that cortical modulations and 
interneuronal correlations initially increase during BMI operation, 
which have been suggested to serve a role in the initial learning of 
unfamiliar tasks. Additionally, we have reported a decrease in variance 
associated with neuronal modulations during BC without arms learn- 
ing (40). Still, the potential function served by the decrease in neuronal 
correlations observed in our study and others (42, 43) remains to be 
fully understood. The most basic interpretation of this result is that 
correlated activity benefits early learning, but firing rate independence 
sustains later stages of the learning process when independent control 
of both arms is learned. Notably, changes were observed in a multi- 
tude of fronto-parietal structures across both cortical hemispheres. 

Previously, we reported elevated correlated activity between neu- 
rons (3) and increased neuronal modulations unrelated to movement 
kinematics (40) during early sessions of unimanual BC without arms. 
A similar change in neural correlations was reported in human electro- 
encephalogram (EEG) studies in which interhemispheric EEG co- 
herence decreased during bimanual task learning (42, 43). Thus, our 
monkey data indicate that inter-area and interhemispheric correlations 
may transiently increase during the initial learning phase and then de- 
crease after subjects have perfected bimanual motor behavior. 

Our findings demonstrate a BMI that controls movements of two 
limbs in real time, using neuronal ensemble data recorded from both 
cerebral hemispheres. Our data may contribute to the development of 
future clinical neuroprosthetics systems aimed at restoring bimanual 
motor behaviors. A key observation is that the inclusion of two limbs 
within a single BMI decoding paradigm produced the best predictions 
and that this approach demonstrated that both arms could be con- 
trolled independently. The importance of bimanual movements in 
our everyday activities and specialized skills cannot be overstated 
(44). Future clinical applications of BMIs aimed at restoring mobility 
in paralyzed patients will benefit greatly from the incorporation of 
multiple limbs. It still remains to be tested how well BMIs would control 
motor activities requiring precise interlimb coordination. From this dem- 
onstration of BMI control over independent movements in two arms, it is 
clear that performance would benefit from the inclusion of large popula- 
tions of neurons and multiple areas in both hemispheres. 
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Another key finding from the current study is that our bimanual 
BMI allowed four degrees of freedom to be decoded across two limbs. 
In a practical sense, our results suggest that to reproduce complex body 
movements using BMI control, the contributions of separate but func- 
tionally related muscle groups should be modeled jointly, not separately. 
Future studies will have to clarify such conjoint representation for a 
broader range of bimanual movements. 

Another feature of our study with implications for neuroprosthet- 
ics was the utilization of a virtual environment within which a subject 
would control realistic avatar arms both during joystick control and 
brain control tasks. Decoding arm movements from neural activity of 
both monkeys proved robust and persistent across many sessions, 
even when subjects passively observed avatar arm movements (Fig. 
7A). In future clinical BMI applications, particularly in those invol- 
ving patients suffering from devastating levels of paralysis, the use of 
realistic, intuitive virtual limb effectors may become a critical com- 
ponent of the BMI training. In upper limb control, arm-centric spatial 
reference frames play a pivotal role in goal-oriented movements (45, 46) 
and may provide a basis for the perception of body schema (47). Even 
when the arm movements were simply observed, sensorimotor (14) and 
premotor (47, 48) neurons in macaques have been shown to encode limb 
kinematics. Within a BMI context, providing the subject with virtual 
arms, instead of cursor circles, could tap into the existing, arm-centric 
biological framework (45-49) and enhance the process of experience- 
dependent plasticity, which is believed to underlie the mechanism 
through which subjects learn to operate a BMI. Preliminary results from 
our laboratory (Fig. 2) provide an early indication that this difference 
(cursor versus avatar arm) is evident to macaques. Therefore, it could 
be further exploited in both behavioral and brain control research par- 
adigms aimed at enhancing the user’s experience as they learn to op- 
erate a BMI. 

Overall, our study demonstrates that cortical large-scale recordings 
can enable bimanual BMI operations in primates, a type of operation 
that will be necessary for future advanced clinical neural prostheses. 


MATERIALS AND METHODS 


Study design 
The objective of the study was to elucidate key differences in cortical 
control between unimanual and bimanual movements and to imple- 
ment a BMI paradigm based on real-time decoding of large-scale, 
multi-area cortical recordings to produce control of two virtual arms 
simultaneously. Furthermore, a BMI strategy was sought that required 
no movements of the subjects’ own arms during the execution of bi- 
manual movements. The study’s design conformed to the conven- 
tional requirements for neurophysiological studies in nonhuman 
primates. All studies were conducted with approved protocols from 
the Duke University Institutional Animal Care and Use Committee 
and were in accordance with the National Institutes of Health (NIH) 
Guide for the Care and Use of Laboratory Animals. Two rhesus 
monkeys were used and each was recorded for more than 19 days. 
The major findings were replicated in both monkeys for multiple exper- 
imental sessions. Statistical analyses of the data involved parametric and 
nonparametric tests. ANOVA tests were used to analyze the influence 
of multiple factors, followed by the appropriate post hoc comparisons. 
The first monkey (monkey C; female; 6.2 kg) was overtrained for 
12 months on unimanual and bimanual center-out reaching tasks 
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before the implantation surgery and the experiments in the present 
study. The second monkey (monkey M; male; 10.6 kg) was extensively 
trained before this study on a unimanual joystick task performed with 
the left arm, but was never introduced to the bimanual joystick task. 
Monkey C performed. bimanual joystick experiments until reaching 
greater than 95% correct trials on consecutive sessions. Monkey M 
was intentionally naive to the bimanual BMI task before passive ob- 
servation experiments. Monkey C next began BC with arms 
experiments, and such experiments continued for 24 sessions until 
we observed consistent performance exceeding 75% correct. Both 
monkeys participated in four experiments that were exclusively 
passive observation of bimanual avatar movements. For all passive ob- 
servation experiments, both arms were fully restrained. For the final 
passive observation session of each monkey, electromyogram (EMG) 
recordings of both arms were obtained. Next, both monkeys partici- 
pated in BC without arms experiments. Monkey C reached proficiency 
after 9 sessions, and monkey M after 15 sessions. BC without arms 
experiments with fewer than 10 trials where the monkey attended 
to the screen for a minimum of 90% of the trial were designated 
as null performance and not included in subsequent analyses. 


Task design 

The avatar had been previously developed by our research group 
(14) and used for reaching movements by assigning joystick or 
BMI output to the position of the center of each hand (near the 
base of the middle finger). This hand location was also used as 
the reference point to indicate whether the hand was inside/outside 
a target. 

Bimanual joystick control trials began by moving the right and 
left spring-loaded joysticks such that the right and left avatar hands 
were placed inside the right and left center targets, respectively. The 
center targets were 8-cm squares located in the center of the right 
and left sides of the screen. Next, both hands had to simultaneously 
remain inside these center targets for a hold interval randomly 
drawn from a uniform distribution between 400 and 1000 ms. After 
this hold, the center targets disappeared and two reach targets 
appeared on the screen, one on each side of the screen. The objec- 
tive was to reach with both hands to their respective targets. The 
nearest edge of the peripheral target appeared at a fixed distance of 
8 cm either left, right, above, or below the center target. The 
peripheral targets were 8 cm in diameter during joystick control 
and BC with arms experiments and were 10 cm in diameter during 
BC without arms experiments. With four potential locations per 
arm, there were 16 potential left/right combinations. Each combi- 
nation was equally likely and was assigned randomly. Once both 
targets were entered and both were simultaneously held for 100 
ms, a juice reward was given to the monkey. 

In two recording sessions, joystick control performance included 
unimanual left, unimanual right, and bimanual trials. During this 
paradigm, the first 150 trials were bimanual, then 150 unimanual left, 
then 150 unimanual right, then 100 to 200 bimanual trials or until the 
monkey voluntarily declined to continue the task. During unimanual 
trials, only the single working arm was displayed on the computer 
screen for the monkey. The avatar arm and targets continued to ap- 
pear in the same locations, on the corresponding right or left half of 
the screen. The door on the primate chair for the nonworking arm 
was closed such that only the working arm could reach and manipu- 
late the joystick. 
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Passive observation experiment 

During passive observation experiments, the monkey was seated in a 
customized chair, which immobilized both arms and allowed minimal 
movement of lower limbs. Arms were restrained to a foam-padded 
shelf fixed at a comfortable and natural angle in front of the monkey 
(Fig. 6A). The pronated arms were fastened to the shelf using secure 
and foam-padded adjustable straps. Both monkey C and monkey M 
were used for this experimental paradigm, although the observed 
movements for the two differed slightly. Monkey C observed replayed 
arm movements from its previous bimanual center-out joystick con- 
trol session. Monkey M did not have previous joystick control sessions 
on this task and instead observed movements of the avatar limbs 
along ideal trajectories. These automated movements were enacted 
with a realistic distribution of reaction times and peak velocities, as 
well as acceleration profiles. We obtained the distribution of typical 
reaction times of monkey M for each arm from previous unimanual 
data. The automated passive observation movements were initiated 
after a reaction time drawn from a distribution with the same mean 
and variance as the monkey’s own typical reaction time distribution. 
The same steps were taken to obtain realistic mean reach velocities 
from previous unimanual training. Acceleration and deceleration pe- 
riods and the beginning and end of the reach were added to make the 
generated movement look natural and smooth. Passive observation 
trials followed the same task sequence as the bimanual center-out 
joystick control task. The monkey was rewarded when both avatar 
arms moved into and held its corresponding peripheral target. In addi- 
tion to target-based rewards, a smaller juice reward quantity was dispensed 
at random intervals between 2 and 8 s to encourage the monkey to 
look at the screen throughout the trial. 

To track the monkeys’ attention to the task during nonjoystick 
experiments, we implemented an eye-tracking system. A single camera 
above the display screen was used to record the monkey. A modified 
version of the TLD (tracking-learning-detection) tracking algorithm 
(50) was used to track eye position in real time and identify periods 
of the experiment when the monkey did not attend to the screen. In 
addition, persistent attention to the screen was required to receive juice 
rewards throughout the experiment. For offline analysis, trials were 
separated into attended and unattended trials. To be considered an at- 
tended trial, the monkey's eyes must be attending to the screen for a 
minimum of 90% of the trial epoch. 

Both monkey C and monkey M performed four 20- to 30-min 
passive observation sessions. After these experiments, passive observa- 
tion was used on a daily basis for 5 to 7 min at the beginning of each 
experiment to train the decoding model. For both monkeys, the fourth 
passive observation session was identical to the previous three except 
for EMG electrodes placed on the forearm flexor and extensor, biceps, 
and triceps of each arm. 


Brain control experiment 

The first phase of brain control experiments is called brain control 
with arm movements (BC with arms) and was completed by mon- 
key C immediately after bimanual joystick control sessions. The de- 
coding model for BC with arms was trained on data collected during 
5 to 7 min of joystick control trials at the beginning of the experi- 
ment. Next, the model was fit and movements of the avatar right and 
left arm were under the control of the decoding algorithm. The arms 
remained unrestrained, and the hands continued to manipulate the 
joysticks. Monkey C performed 24 BC with arms sessions within a 
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span of 7 weeks. The first 11 BC with arms sessions used different 
model parameters from those used in the final 13 BC with arms ses- 
sions (see “Neural decoding” section). 

The latter phase of brain control, brain control without arm move- 
ments (BC without arms), began after both monkey C and monkey M 
completed the four passive observation-only experiments. Monkey M 
began BC without arms immediately after session 4, but monkey C 
began about 4 weeks after day 1 of passive observation. During BC with- 
out arms, the monkeys’ arms were restrained in the same way as for 
passive observation. The decoding model for BC without arms was re- 
trained each session with 5 to 7 min of passive observation trials. The 
observed movements of this initial 5- to 7-min training window were gen- 
erated in the same way as those in passive observation—only experiments. 
Next, the model was fit and the movements of the avatar were controlled 
by the decoding algorithm. The major difference with BC without arms 
from BC with arms was that during the brain control phase, the arms 
were completely restrained and covered. 


Surgery and electrode implantation 

Monkey C was implanted about 3 months before the beginning of the 
experiments of this study. Eight 96-channel multielectrode microwire 
arrays (768 total channels) were implanted into bilateral SMA, M1, S1, 
PMd, and PPC using previously described surgical methods (20, 51). 
Electrode arrays were organized as 4 x 10 grids, with each shaft com- 
posed of two or three polyimide-insulated stainless steel microwire 
electrodes with exposed tips. Within the same shaft, three microwires 
were staggered by 400 um in depth. Lateral spacing between shafts 
was 1 mm center to center. Microelectrodes were lowered using inde- 
pendent microdrives such that the deepest microwire was 2 mm below 
the cortical surface. Only 384 of the 768 channels were recorded at a 
given time because of recording system limitations at the time. PMd 
arrays resulted in a low number of quality units and therefore were 
not included in the 384 selected channels. 

Monkey M was implanted 42 months before the beginning of the 
experiments of this study. Arm and leg M1 and S1 were implanted in 
both hemispheres for a total of eight areas, each with 48 electrodes. 
During experiments, all 384 channels were recorded; however, only 
arm M1/S1 channels were used for both online and offline predictions. 
Each 48-channel implant was composed of a 4 x 4 grid, with three 
electrodes per cannula staggered at 400 um. 


Neurophysiological analysis of joystick control trials 

Modulations in cortical neuronal activity were analyzed using PETHs. 
Spike timestamps for each neuron were first put into 50-ms bins, and 
the activity of each neuron was normalized by subtracting the mean 
bin count and dividing by the SD, which is statistically equivalent to 
the z score. This normalization transforms the activity of each neuron 
to represent modulations as the fraction of overall modulations. Single- 
neuron PETHs were computed in terms of firing rate (Hz); however, 
population analysis used the normalized mean firing rate to facilitate 
comparison between neurons with different baseline firing rates. After 
the normalized mean firing rate was computed for each neuron, event- 
related modulations were analyzed by constructing PETHs. Movement- 
related modulations were computed as the difference between normalized 
mean firing rate during the typical movement epoch (150 to 600 ms 
poststimulus) and normalized mean firing rate during the baseline 
epoch (600 to 100 ms prestimulus). This represents a difference in z 
scores and is referred to as Az in the analysis of this study. We computed 
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Az on a single-trial basis and fit multiple linear regression models to 
compute parameters of directional tuning: 


Az = AL, + B*Ly + C#R, + DAR, (1) 
where (L,, Ly) and (R,, Ry) are («y) positions of the left and right tar- 
gets on each trial. Coefficients A, B, C, and D were fit for each neuron 
with regression. Preferred direction of each hand for each neuron was 
computed using vectors (4;B’) and (C.D). 

A second metric based on Az was used to evaluate depth of direction- 
al tuning. A mean Az was computed for each neuron, for each direction. 
Tuning depth was measured as the SD of the mean Az distribution across 
different directions. Overall movement-related modulations were esti- 
mated as the mean of Az absolute value. 

For comparisons between unimanual and bimanual modulation, 
all data were collected within the same session. The bimanual and uni- 
manual PETHs shown in Fig. 3 represent neural activity normalized 
by the same mean and SD of that single session. Within unimanual 
trials, Eq. 1 was modified to only compute coefficients reflecting mod- 
ulations of the working arm. 


Neurophysiological analysis of passive observation 

and brain control 

Similar analyses were also applied to passive observation and brain 
control experiment data. Initially, PETHs aligned on target presen- 
tation, as described previously, were computed. From the PETH, the 
modulation strength Az was computed for each area and for each of 
the 16 potential movement directions. In addition, we computed Az in 
this way for each passive observation experiment day to observe the 
short- and long-term changes in this parameter. When computed as 
the normalized mean firing rate (fig. $4), this metric gives an estima- 
tion of the population response amplitude as a z score, thus facilitating 
comparisons across sessions regardless of the persistence of all recorded 
units from the previous session. 

A second neurophysiological analysis used during passive observa- 
tion and brain control experiments was neuronal spike count correla- 
tion. Binned (25-ms bin size) neural activity patterns of single neurons 
of full experiments (not single trials, as in PETH analysis) were com- 
pared against the activity profiles of all other neurons in the population 
in a pairwise fashion. The comparison between neuron activity profiles 
was quantified as the Pearson’s correlation coefficient r between equal 
length time series a and b as in Eq. 2: 


Yi-1(ai - 2) (b; - 5) 
Dia - a)? fea (b; = by 


where 7 is the length of a and b. Spike count correlation was quantified 
as the mean r of all pairwise comparisons, quantified as in Eq. 3: 


r(a, b) = corr(a, b) = 


(2) 


F = mean(Vn; € Y, Vn; € Y, corr(nj, n;)) (3) 


where ‘¥ is the ensemble of all recorded neurons and corr(nj,, nj) is 
defined in Eq. 2. We extended this analysis to identify correlations 
within a cortical area (for example, an M1 neuron correlated with 
another M1 neuron) as well as between areas (for example, M1 neu- 
ron and SMA neuron), as well as intrahemisphere versus interhemi- 
sphere. Only cortical activity during periods of the session when the 
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monkey was attending to the screen was used for spike count corre- 
lation comparisons. 


Neural decoding 

Right and left arm kinematics were decoded using a UKF as described 
elsewhere (24). The same model was used for both offline and online 
predictions with three past taps and two future taps of 100-ms binned 
neural activity. During the first 11 sessions of BC with arms, a first- 
order UKF was used with only a single past tap of neural activity (Fig. 
6C, red circles). Later, BC with arms sessions and all BC without arms 
sessions used a fifth-order UKF. These settings were determined em- 
pirically by offline comparison (fig. S3), as well as from previous stu- 
dies in our laboratory (14). Offline predictions were computed for 
both active and passive observation experiments in a similar way. Off- 
line, we used 6 min of neural data collected during attempted trials 
(for joystick control) or while attending to the screen (passive obser- 
vation) to fit the UKF tuning model. For unimanual offline analysis, 
the 2D tuning model fit binned neural activity y(t) as a function of 
single-arm (x,y) position, velocity, and quadratic terms of both as 
in Eq. 4: 


s,(t) 


y(t) =[b, by] be | +b pos, (t)” 4 pos, (t)” + 


[bu bs]] Very) | + ooy/veb (+ veh 


(4) 


For bimanual experiments, both joystick control and passive observa- 
tion, a 4D extension was made to the quadratic tuning model of Eq. 4. 
More specifically, y(t) was now formulated as a function of bimanual (x,y) 
positions, velocities, and quadratic terms of each as in Eq. 5: 


pos, (4) 
y(t) =[b, by b3 ba] a im 
pos, p(t) 
[bs be] 1 pos.z( pos, i (t) 4 
y os.e(0)* + ose) 
vel,z(t) 
be bo alc | * 
velyp(t) 
[bu dv] vely(t)~ + velyz () . 
V velya(t)” + velyr(t) 


We performed several preliminary analyses to optimize the amount of 
training data required for each model and generally found that 5 to 7 
min was sufficient and, beyond this, yielded marginal improvements. Pre- 
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diction performance was measured using correlation coefficient r. For 
each analysis, r was computed five times per condition and averaged, 
using a k-fold cross-validation technique. Hence, we report prediction 
rasa distribution (mean + SEM) rather than a single point. We opted to 
use ras the metric for prediction accuracy because of its common usage 
in other BMI studies. Offline predictions using EMG activity were per- 
formed using a similar procedure. Eight EMG voltage channels were 
each resampled at 10 Hz—the same rate as used for neural decoding. 
All other decoding steps and models (Eq. 5) were equivalent for the two 
methods. 

In addition to computing r using all recorded neurons, we com- 
puted random neuron-dropping curves to evaluate the functional ef- 
fect of number of recorded neurons on offline prediction performance. 
This analysis was conducted separately for unimanual and bimanual 
conditions, although the amount of training data was enforced to be 
equal for both analyses. For each neuron-dropping curve, the number 
of neurons was increased at fixed intervals. At each neuronal quantity, 
n randomly selected neurons were used to both train the model and 
make predictions of bimanual kinematics on a separate block of test 
data. This procedure was repeated five times at each step, each time a 
random subset of neurons was selected and a different block of the 
session was designated as training data to cross-validate our results. 
In addition, neuron-dropping curves were computed both overall 
and by cortical area (Figs. 5, A to D, and 6G). 

When the UKF model was fit, both offline and online, one of the 
parameters computed was the noise covariance matrix R. This R matrix 
was estimated using the product of the regression residual matrix and 
its transpose and was normalized by several constant terms (24). Fur- 
thermore, the diagonal of R has a practical interpretation: values closer 
to 1 indicate that the neuron has a contribution that could largely be 
accounted for simply by noise. Conversely, lower values along the R- 
diagonal contribute more to the fitting of the neural tuning model. 
We use this R-diagonal term for each neuron as a proxy for its involve- 
ment in representing movements of a single arm. By computing R for 
right and left arms within a single experiment, we were able to evaluate 
the multifunctionality of a single neuron, and how this property was 
affected by bimanual modes of movements (fig. $2, B and C). 

Online predictions using the UKF were computed using the same 
model as was used for offline analysis. Neuronal timestamp data from 
each of the three acquisition systems were sent over the local network 
to a single master computer. UKF algorithms within the custom BMI 
suite decoded the activity of all recorded (monkey C) or all arm area 
(monkey M) neurons. Output from the single 4D UKF assigned (x,y) 
position of both right and left hands of the avatar. 

To generate an estimate of the chance performance level during brain 
control experiments, each session was replayed offline with the pe- 
ripheral target locations selected from a randomly shuffled array of 
the session’s actual target location combinations. Each trial proceeded 
according to the same contingencies as during online BMI control in- 
cluding the 10-s timeout period and target hold times. If each virtual 
hand passed into and held the center targets, then moved into and 
held the peripheral targets at random locations, then the trial was cor- 
rect. This procedure was repeated 10 times for each BC with arms 
(monkey C) and BC without arms session (both monkeys). 

A discrete classifier was used to make predictions of target location 
from both passive observation and BC without arms sessions. We 
used a k-NN model with k = 5 for these predictions. k-NN is a non- 
parametric classification algorithm that searches nearest-neighbor data 
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(from training data) within the feature space—in the case of neural 
predictions, this would be a space defined by binned firing rates of 
all the neurons. By searching for the k-NNs to the test data, the al- 
gorithm can be made more robust. The classifications of each “neigh- 
bor” are counted as votes toward the prediction of the unknown sample 
class. Neural activity was binned into a single 250-ms window during a 
specified epoch aligned on target onset. We then slid this window at 
25-ms increments along the task interval from —0.5 to 0.75 s, and at 
each step, the k-NN model was fit with 75% of the session’s trials and 
target location predictions were made on the remaining 25% of trials. 
Train and sample trials were randomly selected five times, and the 
resulting prediction performance in terms of fraction correct predic- 
tion was averaged. In addition to the sliding window k-NN analysis, 
we used k-NN to generate a neuron-dropping curve based on activity 
during BC without arms. Similar to the continuous UKF model, we gen- 
erated predictions of target locations using a varying number of neurons, 
ranging from 1 to all available. Predictions for the 50- to 500-ms window 
were compiled, and the most common output was selected as the “vote.” 
Each test trial therefore had one vote, rather than computed performance 
as a function of time. Performance was computed in terms of fraction 
correct as a function of population size. 


Statistical analysis 

Several statistical methods were used in this study to validate the re- 
sults we obtained. We used a ft test (a = 0.05) for parametric tests and 
the Wilcoxon signed rank test (a = 0.05) or Wilcoxon rank sum test 
(a. = 0.05) for nonparametric tests when data were not drawn from a 
normal distribution. Both one-way and two-way ANOVA tests (a = 
0.05) were used in the statistical analysis of correlated neural activity of 
Fig. 7. k-NN classifier performance (Fig. 6) was measured as fraction 
correct prediction. In this analysis, chance-level performance was '/4. 
The 95% confidence interval was constructed using the one-proportion 
z test (Eq. 6): 


_ P-Po ; 
a= 6) 


where 71 is the number of trials used for test data and pp = 0.25. Offline 
predictions of cursor X and Y position were compared with actual 
cursor positions using Pearson’s product-moment correlation coef- 
ficient r. To generate a distribution of r, we repeated the prediction 
using a five-fold cross-validation technique, with each iteration using 
a different block of data for training and test data. 


SUPPLEMENTARY MATERIALS 


www.sciencetranslationalmedicine.org/cgi/content/full/5/210/210ra154/DC1 
Fig. S1. Behavioral results from bimanual and unimanual joystick experiments. 
Fig. $2. Decoding performance during joystick control. 

Fig. S3. Improvements to UKF model. 

Fig. $4. Plasticity during passive observation training. 

Fig. S5. Arm EMGs during passive and active trials. 

Fig. S6. Temporal changes in prediction accuracy during passive observation. 
Table $1. Unimanual and bimanual modulation differences by area. 

Movie $1. Screen capture of bimanual center-out joystick control trials. 

Movie S2. Video of monkey performing bimanual joystick control trials using two joysticks. 
Movie $3. Video of monkey performing bimanual BC with arms. 

Movie $4. Screen capture of bimanual BC without arms. 


Vol 5 Issue 210 210ra154 12 


415 


RESEARCH ARTICLE 


REFERENCES AND NOTES 


1. 


20. 


21. 


22. 


23. 


24. 


25. 


26. 


27. 


28. 


S. P. Swinnen, J. Duysens, Neuro-Behavioral Determinants of Interlimb Coordination : A Multi- 
disciplinary Approach (Kluwer Academic, Boston, MA, 2004). 


. M. A. Lebedev, M. A. Nicolelis, Brain-machine interfaces: Past, present and future. Trends 


Neurosci. 29, 536-546 (2006). 


. M. A. Nicolelis, M. A. Lebedev, Principles of neural ensemble physiology underlying the 


operation of brain-machine interfaces. Nat. Rev. Neurosci. 10, 530-540 (2009). 


. M.A. Lebedev, M. A. Nicolelis, Toward a whole-body neuroprosthetic. Prog. Brain Res. 194, 


47-60 (2011). 


. C. A. Chestek, J. P. Cunningham, V. Gilja, P. Nuyujukian, S. |. Ryu, K. V. Shenoy, Neural 


prosthetic systems: Current problems and future directions. Conf. Proc. IEEE Eng. Med. Biol. 
Soc. 2009, 3369-3375 (2009). 


. P. G. Patil, J. M. Carmena, M. A. Nicolelis, D. A. Turner, Ensemble recordings of human 


subcortical neurons as a source of motor control signals for a brain-machine interface. 
Neurosurgery 55, 27-35 (2004). 


. J. Wessberg, C. R. Stambaugh, J. D. Kralik, P. D. Beck, M. Laubach, J. K. Chapin, J. Kim, S. J. Biggs, 


M. A. Srinivasan, M. A. Nicolelis, Real-time prediction of hand trajectory by ensembles of cor- 
tical neurons in primates. Nature 408, 361-365 (2000). 


. J. M. Carmena, M. A. Lebedev, R. E. Crist, J. E. O'Doherty, D. M. Santucci, D. F. Dimitrov, P. G. Patil, 


C.S. Henriquez, M. A. Nicolelis, Learning to control a brain-machine interface for reaching and 
grasping by primates. PLoS Biol. 1, E42 (2003). 


. M. D. Serruya, N. G. Hatsopoulos, L. Paninski, M. R. Fellows, J. P. Donoghue, Instant neural 


control of a movement signal. Nature 416, 141-142 (2002). 


. M. A. Lebedev, J. M. Carmena, J. E. O'Doherty, M. Zacksenhouse, C. S. Henriquez, J. C. Principe, 


M. A. Nicolelis, Cortical ensemble adaptation to represent velocity of an artificial actuator 
controlled by a brain-machine interface. J. Neurosci. 25, 4681-4693 (2005). 


. J. E. O'Doherty, M. A. Lebedev, T. L. Hanson, N. A. Fitzsimmons, M. A. Nicolelis, A brain-machine 


interface instructed by direct intracortical microstimulation. Front. Integr. Neurosci. 3, 20 (2009). 


. M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, A. B. Schwartz, Cortical control of a 


prosthetic arm for self-feeding. Nature 453, 1098-1101 (2008). 


. L. R. Hochberg, D. Bacher, B. Jarosiewicz, N. Y. Masse, J. D. Simeral, J. Vogel, S. Haddadin, J. Liu, 


S. S. Cash, P. van der Smagt, J. P. Donoghue, Reach and grasp by people with tetraplegia using 
a neurally controlled robotic arm. Nature 485, 372-375 (2012). 


. J. E. O'Doherty, M. A. Lebedev, P. J. Ifft, K. Z. Zhuang, S. Shokur, H. Bleuler, M. A. Nicolelis, 


Active tactile exploration using a brain-machine-brain interface. Nature 479, 228-231 (2011). 


. U. Rokni, O. Steinberg, E. Vaadia, H. Sompolinsky, Cortical representation of bimanual 


movements. J. Neurosci. 23, 11577-11586 (2003). 


. O. Steinberg, O. Donchin, A. Gribova, S. Cardosa de Oliveira, H. Bergman, E. Vaadia, Neuronal 


populations in primary motor cortex encode bimanual arm movements. Eur. J. Neurosci. 15, 
1371-1380 (2002). 


. J. Tanji, K. Okano, K. C. Sato, Neuronal activity in cortical motor areas related to ipsilateral, con- 


tralateral, and bilateral digit movements of the monkey. J. Neurophysiol. 60, 325-343 (1988). 


. O. Donchin, A. Gribova, O. Steinberg, H. Bergman, E. Vaadia, Primary motor cortex is 


involved in bimanual coordination. Nature 395, 274-278 (1998). 


. O. Kazennikov, B. Hyland, M. Corboz, A. Babalian, E. M. Rouiller, M. Wiesendanger, Neural 


activity of supplementary and primary motor areas in monkeys and its relation to bi- 
manual and unimanual movement sequences. Neuroscience 89, 661-674 (1999). 

M. A. Nicolelis, D. Dimitrov, J. M. Carmena, R. Crist, G. Lehew, J. D. Kralik, S. P. Wise, Chronic, 
multisite, multielectrode recordings in macaque monkeys. Proc. Natl. Acad. Sci. U.S.A. 100, 
11041-11046 (2003). 

|. H. Stevenson, K. P. Kording, How advances in neural recording affect data analysis. 
Nat. Neurosci. 14, 139-142 (2011). 

S. Shokur, J. E. O'Doherty, J. A. Winans, H. Bleuler, M. A. Lebedev, M. A. Nicolelis, Expanding 
the primate body schema in sensorimotor cortex by virtual touches of an avatar. Proc. Natl. 
Acad. Sci. U.S.A. 110, 15121-15126 (2013). 

J. L. Collinger, B. Wodlinger, J. E. Downey, W. Wang, E. C. Tyler-Kabara, D. J. Weber, A. J. McMorland, 
M. Velliste, M. L. Boninger, A. B. Schwartz, High-performance neuroprosthetic control by an 
individual with tetraplegia. Lancet 381, 557-564 (2013). 

Z. Li, J. E. O'Doherty, T. L. Hanson, M. A. Lebedev, C. S. Henriquez, M. A. Nicolelis, Unscented 
Kalman filter for brain-machine interfaces. PLoS One 4, e6243 (2009). 

D. M. Taylor, S. |. Tillery, A. B. Schwartz, Direct cortical control of 3D neuroprosthetic de- 
vices. Science 296, 1829-1832 (2002). 

L. R. Hochberg, M. D. Serruya, G. M. Friehs, J. A. Mukand, M. Saleh, A. H. Caplan, A. Branner, 
D. Chen, R. D. Penn, J. P. Donoghue, Neuronal ensemble control of prosthetic devices by a 
human with tetraplegia. Nature 442, 164-171 (2006). 

R. Wahnoun, J. He, S. |. Helms Tillery, Selection and parameterization of cortical neurons for 
neuroprosthetic control. J. Neural Eng. 3, 162-171 (2006). 

S. P. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, G. M. Friehs, M. J. Black, Point-and-click 
cursor control with an intracortical neural interface system by humans with tetraplegia. 
IEEE Trans. Neural Syst. Rehabil. Eng. 19, 193-203 (2011). 


www.ScienceTranslationalMedicine.org 6 November 2013 


29. 


30. 


31. 


32. 


33. 


34. 


35. 


36. 


37. 


38. 


39. 


40. 


41. 


M. Hauschild, G. H. Mulliken, |. Fineman, G. E. Loeb, R. A. Andersen, Cognitive signals for 
brain-machine interfaces in posterior parietal cortex include continuous 3D trajectory 
commands. Proc. Natl. Acad. Sci. U.S.A. 109, 17075-17080 (2012). 

N. Hatsopoulos, J. Joshi, J. G. O'Leary, Decoding continuous and discrete motor behaviors 
using motor and premotor cortical ensembles. J. Neurophysiol. 92, 1165-1174 (2004). 

H. Scherberger, R. A. Andersen, Target selection signals for arm reaching in the posterior 
parietal cortex. J. Neurosci. 27, 2001-2012 (2007). 

D. M. Santucci, J. D. Kralik, M. A. Lebedev, M. A. Nicolelis, Frontal and parietal cortical 
ensembles predict single-trial muscle activity during reaching movements in primates. 
Eur. J. Neurosci. 22, 1529-1540 (2005). 

J. Requin, J. C. Lecas, N. Vitton, A comparison of preparation-related neuronal activity changes 
in the prefrontal, premotor, primary motor and posterior parietal areas of the monkey cortex: 
Preliminary results. Neurosci. Lett. 111, 151-156 (1990). 

D. J. Saha, P. Morasso, Stabilization strategies for unstable dynamics. PLoS One 7, €30301 (2012). 
C. E. Peper, R. G. Carson, Bimanual coordination between isometric contractions and rhythmic 
movements: An asymmetric coupling. Exp. Brain Res. 129, 417-432 (1999). 

A. Ridderikhoff, C. L. Peper, P. J. Beek, Unraveling interlimb interactions underlying bi- 
manual coordination. J. Neurophysiol. 94, 3112-3125 (2005). 

N. A. Fitzsimmons, M. A. Lebedev, |. D. Peikon, M. A. Nicolelis, Extracting kinematic parameters 
for monkey bipedal walking from cortical neuronal ensemble activity. Front. Integr. Neurosci. 3, 
3 (2009). 

S. |. Tillery, D. M. Taylor, Signal acquisition and analysis for cortical control of neuropros- 
thetics. Curr. Opin. Neurobiol. 14, 758-762 (2004). 

A. P. Georgopoulos, R. E. Kettner, A. B. Schwartz, Primate motor cortex and free arm move- 
ments to visual targets in three-dimensional space. Il. Coding of the direction of movement by 
a neuronal population. J. Neurosci. 8, 2928-2937 (1988). 

M. Zacksenhouse, M. A. Lebedev, J. M. Carmena, J. E. O'Doherty, C. Henriquez, M. A. Nicolelis, 
Cortical modulations increase in early sessions with brain-machine interface. PLoS One 2, e619 
(2007). 

B. Olson, J. Si, Evidence of a mechanism of neural adaptation in the closed loop control of 
directions. Int. J. Intell. Comput. Cybern. 3, 5-23 (2008). 


. C. Gerloff, F. G. Andres, Bimanual coordination and interhemispheric interaction. Acta Psychol. 


110, 161-186 (2002). 


. F. G. Andres, T. Mima, A. E. Schulman, J. Dichgans, M. Hallett, C. Gerloff, Functional coupling of hu- 


man cortical sensorimotor areas during bimanual skill acquisition. Brain 122 (Pt. 5), 855-870 (1999). 


. S. P. Swinnen, N. Wenderoth, Two hands, one brain: Cognitive neuroscience of bimanual 


skill. Trends Cogn. Sci. 8, 18-25 (2004). 


. C.L. Colby, Action-oriented spatial reference frames in cortex. Neuron 20, 15-24 (1998). 
. G. Rizzolatti, L. Fadiga, L. Fogassi, V. Gallese, The space around us. Science 277, 190-191 (1997). 
. M. S. Graziano, Where is my arm? The relative role of vision and proprioception in the neu- 


ronal representation of limb position. Proc. Natl. Acad. Sci. U.S.A. 96, 10418-10421 (1999). 


. T. Ochiai, H. Mushiake, J. Tanji, Effects of image motion in the dorsal premotor cortex 


during planning of an arm movement. J. Neurophysiol. 88, 2167-2171 (2002). 


. M. S. Graziano, C. G. Gross, A bimodal map of space: Somatosensory receptive fields in the 


macaque putamen with corresponding visual receptive fields. Exp. Brain Res. 97, 96-109 (1993). 


. Z. Kalal, K. Mikolajczyk, J. Matas, Tracking-learning-detection. /EEE Trans. Pattern Anal. 


Mach, Intell. 34, 1409-1422 (2012). 


. G. Lehew, M. A. L. Nicolelis, State-of-the-art microwire array design for chronic neural 


recordings in behaving animals, in Methods for Neural Ensemble Recordings, M. A. L. Nicolelis, 
Ed. (CRC Press, Boca Raton, FL, ed. 2, 2008). 


Acknowledgments: We are grateful for the work of G. Lehew and J. Meloy in the building of the 
experimental setup and the multielectrode arrays, D. Dimitrov and L. Oliveira for conducting neu- 
rosurgeries, T. Phillips for experimental support, and S. Halkiotis for administrative support. 
Funding: This work was supported by NIH grant DP1MH099903 and NIH (National Institute of 
Neurological Disorders and Stroke) RO1INSO73952 awarded to M.A.L.N. The content is solely the 
responsibility of the authors and does not necessarily represent the official views of the Office of 
the Director or NIH. Author contributions: M.A.L.N.,, MA.L, PJ. and S.S. designed the study. PJ.I. 
performed all of the joystick and brain control experiments. PJ.I. and S.S. performed all of the 
passive observation experiments. Z.L. and PJ.l. wrote decoding software for the custom BMI suite 
used for experiments. PJ. S.S., M.A.L, and M.A.L.N. analyzed the data and wrote the manuscript. 
Competing interests: The authors declare that they have no competing interests. 


Submitted 18 March 2013 
Accepted 18 October 2013 
Published 6 November 2013 
10.1126/scitranslmed.3006159 


Citation: P. J. Ifft, S. Shokur, Z. Li, M. A. Lebedev, M. A. L. Nicolelis, A brain-machine interface 
enables bimanual arm movements in monkeys. Sci. Transl. Med. 5, 210ra154 (2013). 


Vol 5 Issue 210 210ra154 13 


416 


SCIENTIFIC REPLIRTS 


OPEN Wireless Cortical Brain-Machine 
Interface for Whole-Body 
Navigation in Primates 


Received: 23 October 2015 ° Sankaranarayani Rajangam?2*, Po-He Tseng?", AllenYin23, Gary Lehew?2, 
Accepted: 09 February 2016 - David Schwarz™2, Mikhail A. Lebedev & Miguel A. L. Nicolelis? 23,45 
Published: 03 March 2016 : 

Several groups have developed brain-machine-interfaces (BMIs) that allow primates to use cortical 

: activity to control artificial limbs. Yet, it remains unknown whether cortical ensembles could represent 

: the kinematics of whole-body navigation and be used to operate a BMI that moves a wheelchair 

: continuously in space. Here we show that rhesus monkeys can learn to navigate a robotic wheelchair, 
using their cortical activity as the main control signal. Two monkeys were chronically implanted with 
multichannel microelectrode arrays that allowed wireless recordings from ensembles of premotor and 
sensorimotor cortical neurons. Initially, while monkeys remained seated in the robotic wheelchair, 
passive navigation was employed to train a linear decoder to extract 2D wheelchair kinematics from 
cortical activity. Next, monkeys employed the wireless BMI to translate their cortical activity into the 

: robotic wheelchair’s translational and rotational velocities. Over time, monkeys improved their ability 

: to navigate the wheelchair toward the location of a grape reward. The navigation was enacted by 

: populations of cortical neurons tuned to whole-body displacement. During practice with the apparatus, 
we also noticed the presence of a cortical representation of the distance to reward location. These 
results demonstrate that intracranial BMIs could restore whole-body mobility to severely paralyzed 
patients in the future. 


The wheelchair remains the main device to assist navigation in people with motor disabilities, particularly those 
suffering from severe cases of body paralysis!. Up to date, noninvasive BMI approaches, mostly based on electro- 
encephalography (EEG), have been dominant in assistive devices that enable direct brain control over navigation 
in a powered wheelchair?°. Although intracranial BMIs hold promise to offer a superior performance over non- 
invasive systems®’ and approximately 70% of paralyzed patients are willing to accept surgically implanted elec- 
trodes in their brains to gain control over their assistive devices*”, only a few studies have previously attempted to 
apply intracranial BMIs to wheelchair control'®”’. 

One design for an intracranial BMI that controls wheelchair movements is based on joystick movement. This 
design was previously introduced by our laboratory’® and by Xu ef al.'!. In our experiments, rhesus monkeys were 
housed in a home cage equipped with a joystick. They used the joystick to continuously control the movements 

: of an external wheelchair. This control was also performed in BMI mode, where the wheelchair coordinates 

: were extracted from cortical neuronal ensembles”®. In some BMI experiments the monkeys continued to assist 

: movement with the joystick, in the others the joystick was removed and the monkeys controlled the wheelchair 

: without producing overt arm movements. Recently, Xu et al. described how one monkey seated in a wheelchair"! 

: was trained to move a joystick in one of four directions to set the direction of wheelchair movement. Under BMI 

: control, this monkey continued to move the joystick, but the control signal was derived from a discrete classifier 
that extracted joystick position from cortical activity. 

These BMI designs are similar to those previously used to reproduce arm movements, only replacing the 
end actuator with a robotic wheelchair. However, such BMIs cannot be used by severely paralyzed quadriple- 
gic patients, who cannot produce arm movements. Additionally, neural control of arm movements may be an 
unnecessary intermediary to enable wheelchair navigation. Here we hypothesized that kinematics of whole-body 
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movements could be extracted directly from sensorimotor cortical ensembles, and utilized to control a BMI for 
wheelchair control. As a first step towards the development of a clinically relevant device of this type, we utilized 
large scale recordings from multiple cortical areas’ obtained by our recently developed multichannel wireless 
recording system” to enable BMI control over whole-body navigation in a robotic wheelchair. 


Results 

The study was conducted in two monkeys (K and M) chronically implanted with multielectrode arrays in multiple 
cortical areas in both hemispheres (Fig. 1). Neuronal ensemble activity was sampled using our 512-channel wire- 
less recording system!°. In Monkey K, 79 neurons were recorded bilaterally in the primary motor cortex (M1), 35 
in the right primary somatosensory cortex (S1), and 26 in the right dorsal premotor cortex (PMd). In Monkey M, 
72 neurons were recorded in bilateral M1, and 72 in bilateral S1. 

Monkeys were seated in a mobile robotic wheelchair (Fig. 1). Each trial started with the robot being placed in 
one of three starting locations. A food reward (grape) was then dispensed onto a plate mounted at the target loca- 
tion, 1.9-2.1 m away. The food location was constant for all trials. Monkeys navigated from the starting location 
to the food, which completed the trial. At the beginning of each recording session we ran passive navigation trials, 
with the robot's routes preprogrammed and identical from day to day (Fig. S1, Movie S1). The passive trajectories 
were empirically chosen based on two principles. First, we wanted monkeys to perceive that these trajectories, 
although noisy, clearly headed to the grape dispenser. Accordingly, the training routes were constructed from 
straight lines toward the grape dispenser, which were distorted by moderate over-turning and correcting-turning. 
Second, we wanted to enrich the training routes by backwards movements, which monkeys could use to correct 
navigation errors. Such backward movements were added. Cortical neuronal responses to passive whole-body 
displacements were utilized to obtain initial settings for two Wiener filters”, which decoded translational 
(R of predictions of 0.62 + 0.02 and 0.43 + 0.02 for monkeys K and M, respectively; mean + standard error) and 
rotational velocity (0.30 + 0.02 and 0.22 + 0.02) from neuronal ensemble activity (Fig. $2, as an example). Note 
that both translational and rotational velocities were expressed in chair-centered coordinates, not in room coor- 
dinates. Accordingly, this control mode was largely independent and unrelated to the locations of objects in 
the room, such as the food dispenser (see below). After the decoders were trained, the mode of operation was 
switched to a direct brain-control mode, where the outputs of the Wiener filters operated the wheelchair in real 
time. As shown below, monkeys were able to learn how to use their brain-derived signals to drive the wheelchair 
even though the decoding of wheelchair velocity was not superb during the passive navigation training period. 


Neuronal tuning to translational and rotational velocity. Individual neurons were tuned to the 
wheelchair movements during both passive and BMI navigation (Tuning Depth of 0.24 + 0.005 and 0.18 + 0.002 
for passive movements, for monkeys K and M, respectively; and 0.15 + 0.005 and 0.12 + 0.003 for BMI control; 
see Supplementary Methods). Color plots of Fig. 2A show tuning patterns to translational and rotational velocity 
for two representative neurons (neurons A and B), both recorded in Monkey K’s M1. Again, these plots refer to 
wheelchair-centered coordinates, not to room coordinates. During passive navigation, neuron A increased its fir- 
ing rate when the robot moved backward, whereas neuron B increased its firing rate when the robot moved back- 
ward and/or rotated clockwise. During BMI-based navigation, neuron A remained tuned to backward movement, 
whereas neuron B was no longer tuned to backward movements, but remained tuned to clockwise rotations. 


Tuning depth analysis. Following an approach previously introduced by us to analyze the Wiener filter 
performance in BMI control", we calculated tuning depth as a function of time lag between neuronal activity and 
robot movement for the entire sample of recorded neurons in the two monkeys (Fig. 2B). This analysis revealed 
that tuning depth reached its maximum before wheelchair movement during BMI control versus during passive 
navigation. This result was observed for the entire neuronal population recorded in the two monkeys (Fig. 2C, 
Wilcoxon signed rank test, p < 0.01 for both monkeys) and for M1 and S1 populations analyzed separately 
(Wilcoxon signed rank test, p < 0.01 for both monkeys and both regions). This finding likely indicates the differ- 
ence in causal relationships between the two modes of operation: during passive navigation, neuronal responses 
were caused by robot movements, whereas during BMI navigation, robot movements were caused by neuronal 
modulations that preceded the actual robotic wheel chair movement (Fig. $3). Additionally, maximum tuning 
depth occurred earlier for M1 neurons than for $1 neurons during BMI control. This time difference was statis- 
tically significant for Monkey K (— 489.5 ms in M1 vs —431.6 ms in S1; Wilcoxon Rank Sum Test, p < 0.01) and 
marginally missed the 0.05 significance criterion for Monkey M (— 495.7 ms vs — 441.3 ms; p= 0.055). During 
passive movements, the time difference was not significant (p > 0.314). This finding supports previous studies 
that demonstrated a lead of M1 activity compared to $1 during voluntary movements’». Notice, however, that in 
the current study movements were initiated through a BMI without the overt production of body movements. 
Consistent with our previous studies where we compared neuronal tuning characteristics between the period 
when a decoder was trained and the period during BMI operations'*"%, the characteristics of neuronal tuning to 
the wheelchair movements differed between passive navigation and brain control in two ways. First, we observed 
reduction in tuning depth during BMI navigation. However, neurons that were better tuned during passive navi- 
gation were also better tuned during BMI operations, as the tuning depth was actually positively correlated when 
passive navigation was compared with brain control across all neurons (Fig. 2D. Mann-Kendiall Test, p < 0.01). 
Second, tuning diagrams could change substantially for many neurons after the BMI control started (Fig. 2E & 
Fig. S4). In particular, there was a considerable mismatch between the tuning diagrams for neurons that were 
weakly tuned. As shown in Fig. 2E, neurons that were less tuned to velocity commands had lower correlation 
between BMI and passive navigation in their tuning diagrams (calculated as pixel-to-pixel correlation). This 
mismatch was particularly strong for Monkey M, where the mismatch was also clear for the well-tuned neurons 
(Mann-Kendall Test, p = 0.126). The presence of such a mismatch in weakly tuned neurons could reflect the 
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Figure 1. Overview of the experimental design. (A) The mobile robotic wheelchair, which seats a monkey, 
was moved from one of the three starting locations (dashed circles) to a grape dispenser. The wireless 
recording system records the spiking activities from the monkey’s head stage, and sends the activities to 

the wireless receiver to decode the wheelchair movement. (B) Schematic of the brain regions from which 

we recorded units tuned to either velocity or steering. Red dots correspond to units in M1, blue from PMd 
and green from the somatosensory cortex. (C) Three video frames show Monkey K drive toward the grape 
dispenser. The right panel shows the average driving trajectories (dark blue) from the three different starting 
locations (green circle) to the grape dispenser (red circle). The light blue ellipses are the standard deviation of 
the trajectories. 
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Figure 2. Translational and rotational velocity tuning for neurons during passive training and BMI-based 
navigation. (A) Each diagram shows the normalized firing rate as a function of translational and rotational 
velocity commands (bottom-left), where 1 represents the maximum command value sent to the wheelchair 
(positive values represent forward commands for translation, and clock-wise (CW) for rotation). Each sequence 
shows the normalized firing rate at different time lag. Neuron A is tuned to backward movement during both 
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passive training and BMI control. Neuron B is tuned to right turn movement during brain control, but with 
different tuning during passive training. (B) The tuning depth across time for the two neurons in (A). Note that 
the time lag of the peak tuning differs between passive training and BMI control. (C) Distribution of the tuning 
depth to velocity and the lag of peak velocity tuning for all the neural units recorded. The lag of peak velocity 
tuning moved from ~0 ms to ~ —500 ms, i.e. half'a second earlier than the movement onset. (D) Neural units 
that were well tuned during passive navigation, were more likely to be well tuned during BMI navigation. Each 
dot represents one neural unit, and the solid line is the regression line. (E) For units that were better tuned, 
their tuning diagrams between passive and MI navigation were also better correlated in Monkey K, but not in 
Monkey M. 


fact that tuning characteristics during BMI control were particularly dependent on the overall synchrony in the 
neuronal ensemble, the property that changes during BMI learning"®. This interesting finding will be investigated 
further in a future study. In Monkey K, well-tuned neurons tended to retain their tuning properties after the tran- 
sition to BMI control (Mann-Kendall Test, p < 0.01). 


Behavioral improvement overtime. As the training continued for 3 weeks for Monkey M and 6 weeks 
for Monkey K, BMI navigation gradually improved for both monkeys, as evident from the reduction in trial 
duration (Fig. 3A, Mann- Kendall Test, p= 0.025 and 0.045 for Monkey K and M) and trajectory length (Fig. 3B, 
Mann-Kendall Test, p = 0.008 and 0.020 or Monkey K and M). Monkey K started with a 43.1s trial duration 
(median value for the first week), and improved it to 27.3 s (last week). Monkey M improved from 49.1 to 34.7. 
These performance improvements were accompanied by subtle changes in neuronal tuning. We observed that 
the decoder settings gradually changed from session to session. Taking the last recording session as a reference, 
we observed that the decoder settings were very distinct from those in the earlier recording sessions (Fig. 3B, 
Mann-Kendal Test, Monkey K: p = 0.256 and 0.002 for translational and rotational velocity commands; Monkey 
M: p= 0.032 and p = 0.042 for the two commands). This suggests a convergence to a neuronal tuning pattern, 
which was optimal for brain control. 


Inverting decoders decreased performance. After both monkeys reached their best performance levels, 
we tested whether this improvement resulted from directionally tuned neuronal activity or from non-directional 
factors (e.g. temporal patterns of neuronal modulations). To change the BMI directional output without changing 
non-directional components, we reversed the sign of both translational and rotational velocities (ie., forward 
instead of backward movement and counterclockwise rotation instead of clockwise Fig. $5). After this operation, 
navigation accuracy decreased significantly for the two monkeys (Fig. 3C, Wilcoxon signed rank test, p < 0.01 in 
both trial duration and trajectory length for both monkeys). This observation confirmed that monkeys indeed 
learned to use their directionally tuned cortical neurons to navigate the robotic wheelchair and a random walk 
strategy did not yield the desired movement toward the reward location. 


Monkey head orientation. The video tracking of monkey head position showed two basic head orienta- 
tions: (1) toward the food reward, and (2) straight relative to the chair. The first orientation was evident from the 
distribution of monkey head orientation relative to the feeder as the function of trial time (Fig. S6). While the 
monkeys turned their heads in different directions, they looked at the grape feeder more often than at different 
room locations. This tendency was especially prominent in Monkey K who clearly looked at the feeder through- 
out the trial. In both monkeys, head orientation at the feeder became especially prominent when the wheelchair 
arrived at the feeder location. The second type of head orientation, i.e. straight relative to the chair was clear from 
the distribution of head angle relative to the chair (Fig. S7). The tendency to look straight was especially strong in 
Monkey M and less strong in Monkey K who, as shown in the previous analysis, often looked at the food location. 

These basic head orientations were also evident from two correlation analyses. In the first analysis, the angle 
of wheelchair orientation was negatively correlated with the head angle relative to the chair (r = —0.50, p< 0.01 
for Monkey K and r= —0.19, p < 0.01 for Monkey M; angular-angular correlation analysis). In other words, 
monkeys turned their heads in the direction opposite to the chair rotation when the chair turned away from the 
food dispenser. In the second analysis, the angle of wheelchair orientation was positively correlated with the head 
angle in room coordinates (r = 0.61, p < 0.01 for Monkey K and r= 0.83, p < 0.01 for Monkey M; angular-angular 
correlation analysis). This corresponded to the monkeys looking straight relative to the chair. 


Neuronal representation of distance of the wheelchair to the reward location. Although 
BMI decoding did not incorporate absolute position of the robot as a controlled parameter and relied on the 
robot-centered coordinates instead, we observed the presence of positional tuning to reward location in corti- 
cal neurons during the BMI navigation task. This positional tuning is depicted by the peri-distance histograms 
(PDHs) that represent neuronal activity as a function of the distance of the wheelchair to the food dispenser 
(Fig. 4). The positional tuning could not be explained by the monkey arm movements because it occurred before 
the wheelchair reached the docking range, and the monkeys did not move their arms during this period. The 
positional dependence in the population PDHs was specific to the periods of BMI control mode in both mon- 
keys (Fig. 4B, Mann-Kendall trend test, p < 0.01 for both monkeys). No such dependence was observed during 
passive navigation (Fig. 4A). In Monkey K, neuronal firing rates gradually increased as the robot approached the 
feeder (Fig. 4). In Monkey M, there was no such increase until the robot was 0.9 m from the feeder. At this spatial 
position, neuronal firing rates started to increase as the robot approached the feeder (Fig. 4). Although we found 
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Figure 3. Behavioral improvement and the increase of decoder similarities across sessions. (A) Both 
monkeys show significant improvement in the traveling time and distance as they learn. The circles represent 
the median and the error bars show the interquartile range of the medians. (B) Increased correlations between 
decoders trained in earlier sessions and the last session. (left, Monkey K; right, Monkey M (C) Both monkeys 
demonstrate impaired performance once their decoded movement commands were inverted (forward now 
becomes backward, and right turn becomes left turn). The bar graph shows the median and the error bars 
indicate the interquartile range of the medians. 
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Figure 4. Population responses as a function of distance to the reward. (A) Population responses in Monkey 
Kand Mas a function of distance to the reward during passive navigation and BMI navigation (B) Color 
represents normalized firing rate, and the bar graph shows the average unit modulation, calculated by the 
average of the absolute values of all units’ normalized firing rate at each distance. When the cart is getting close 
to the target, the monkeys may reach to the grape, and these reach-related activities are shaded in red. 


the presence of this tuning in brain-control trials, we did not detect the exact instance when this tuning emerged 
during the training period. 

Offline Wiener filter decoders were then trained to predict distance to target from cortical ensemble activity 
for both monkeys. Consistent with the PDHs of Fig. 4, this analysis predicted distance to the reward location from 
neuronal population activity. R of predictions during brain-control were 0.49 + 0.02 and 0.31 + 0.01 for Monkey 
K and Monkey M, respectively. Curiously, while the PDHs for passive navigation mode did not show a steady 
rate of increase, the Wiener filters trained from data within this model also predicted distance to reward (Fig. 4). 
This is because these PDHs still contained neuronal modulations, which the Wiener filter picked and utilized to 
generate predictions. R of predictions (mean + standard error) were 0.42 + 0.02 and 0.22 + 0.02 for Monkey K 
and Monkey M, respectively. However, the filters trained specifically to decode distance for either mode do not 
generalize to the other. Specifically, R of predictions for brain-control mode with filters trained in passive mode 
were 0.05 + /—0.09 and 0.08 + /—0.04; R of predictions for passive mode with filters trained in brain-control were 
0.21 + /—0.12 and 0.24 + /—0.09, for Monkey K and Monkey M, respectively. This observation indicates that the 
distance tuning cannot be attributed to reward related modulation alone. Moreover, cortical neuronal tuning to 
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target distance cannot be explained by the monkeys’ overt reaching movements, since the animals did not pro- 
duce arm movements until the robot arrived at the food dispenser location (Movie S1). Arm reaching for grapes 
did produce neuronal modulations (highlighted in red in Fig. 4), but the comparison of neuronal modulation 
magnitudes during arm movements with those during navigation showed only a weak correlation on a neuron to 
neuron basis (Fig. S8). This indicates that the neuronal firing modulations observed during wheelchair navigation 
could not be explained as being related primarily to the production of arm movements. 


Discussion 

The present study has demonstrated that intracranial wireless cortical ensemble recordings can be harnessed 
to control whole-body navigation in a mobile device such as a robotic wheelchair. Heretofore, intracranial BMI 
research in primates has focused primarily in reconstructing isolated upper limb movements". A few years ago, 
our laboratory introduced the use of BMIs to investigate the possibility of restoring more complex movements, 
such as bipedal locomotion” and then bimanual movements"®. Yet, the investigation of cortical representation of 
whole-body translations, and the likelihood of using intracranial BMIs for wheelchair control, has been almost 
completely neglected. A few primate studies on spatial representation of the environment by hippocampal neu- 
rons!®-”9 have been performed, but these results are not directly applicable to the BMI for wheelchair control. 
Neuronal mechanisms of spatial encoding in rodents have received much more attention”! compared to pri- 
mate studies, but the prospects of BMI navigation based on these neuronal properties was not explored. 

Schwartz et al.'° demonstrated a joystick-based BMI paradigm where monkeys controlled movements of a 
wheelchair by their cortical modulations. The monkeys were not seated in the wheelchair in this study. More 
recently, Xu et al. trained one monkey to steer while seated in a wheelchair using a hand-held joystick to generate 
discrete but not continuous navigation commands". These authors also demonstrated a BMI version of this joy- 
stick control, where hand movements were decoded from M1 ensemble activity to produce steering commands!!. 
This group, however, did not attempt to translate cortical activity directly into whole-body navigation, without 
using hand movements as an intermediary. In another study, monkeys navigated in a virtual environment using a 
joystick while their bodies remained stationary™’. Under these conditions, neuronal firing modulations observed 
in the medial superior temporal cortex depended on whether monkeys actively steered with the joystick or pas- 
sively observed the visual flow of the scene. 

In contrast to these studies, we trained our BMI decoder using a passive wheelchair navigation paradigm. A 
somewhat similar approach was explored in several BMI studies, where subjects passively observed the move- 
ments of an external device and/or imagined voluntarily controlling those movements while the BMI decoder 
was trained'®?>6, This BMI training approach has obvious clinical significance since severely paralyzed subjects 
cannot produce overt body movements to train a BMI decoder. We observed that, after passive navigation was 
employed to set up the decoder parameters, animals significantly improved their navigation performance through 
learning, likely mediated by widespread cortical plasticity’®. 

It is important to emphasize that, different from previous BMI studies in which animals were trained by 
simply observing movements of a virtual actuator on a computer screen (e.g. virtual arm, legs, etc.), here the 
entire animal's body was passively transported by the actuator (wheelchair) during training of the BMI decoder. 
This implies that an integration of vestibular, proprioceptive, visual and auditory inputs influenced the choice 
of our optimal decoder settings, something that had not been tried before in the BMI field. Given the nature 
of our task, it is very likely that multimodal sensory inputs continued to influence neuronal ensemble activity 
during BMI control. In this study, we chose the training routes empirically. In the future, it would be of interest 
to optimize these routes to maximize decoding accuracy. Optimizing the decoder is equivalent to maximizing 
the signal-to-noise ratio (SNR). The SNR could be increased by searching for the wheelchair movement patterns, 
which elicit stronger and more consistent neuronal responses, for example optimal ranges of wheelchair velocities 
and accelerations. In addition to optimizing the passive training procedure, it would be of interest in the future 
to expand the recording areas, so that neural information is recorded from cortical and subcortical areas that 
represent relevant visual and vestibular inputs. Since we recorded mostly from M1 and S1, it is possible that ves- 
tibular and visual signals were underrepresented, and better BMI control could be achieved if we sampled from 
associative areas of parietal and frontal cortices. 

We observed distinct differences in neuronal firing patterns during passive navigation and when monkeys 
were switched to the BMI control phase of our experiments. Most notably, cortical neuronal tuning peaked prior 
to the wheelchair movement during BMI control, likely indicating a causal relationship between cortical activity 
and the type of navigation pattern generated by the animal. During passive navigation, neuronal tuning peaked 
after the wheelchair moved because the cortical neuronal activity was mainly induced by feedback sensory signals 
generated by the passive animal body movements. An approximately 500 ms delay between neuronal modulations 
and the wheelchair response during BMI control is not surprising because there was a mechanical delay to trans- 
late the velocity commands to the wheelchair movement. In the future, optimization strategies for the decoder 
should be explored to account for this delay. In the present study, the 500 ms lag was right in the middle of the 
1000 ms time window used for decoding of neural data, which assured that neural data was collected from the 
most informative epoch. However, some fine adjustments could be helpful in the future, particularly those based 
on adaptive decoders. This heterogeneity in the maximum tuning time concurs with our previous results for BMIs 
aimed at enacting arm" and leg!” movements. In addition to these time lag-dependent changes, learning the BMI 
control evoked changes in directional tuning of cortical neurons to translational and rotational velocity of the 
wheelchair. Based on these findings, we suggest that both temporal and directional characteristics of neuronal 
tuning adapted to improve BMI control. This conclusion is in line with similar findings for upper-limb BMIs!*”. 

We also observed the emergence of a cortical representation of distance to the reward, which was specific 
to the context of the task chosen to test our whole-body BMI. Remarkably, the emergence of distance tuning in 
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cortical neurons was not related to the kinematic variable controlled through the BMI (wheelchair linear and 
rotational velocity encoded in the chair-centered coordinate system). Instead, this additional cortical representa- 
tion apparently emerged during training and become superimposed to the neuronal velocity mapping already 
present in the sensorimotor cortex. The representation of the distance to reward, which also served as a target 
for the wheel chair movement, is likely to be somewhat related to reward encoding by cortical neurons, which 
was recently demonstrated by Marsh et al.”* using an arm-reaching task. Indeed, an increase in neuronal activity 
with the proximity to the reward could indicate a reward-anticipation signal. However, PDHs were very different 
during passive navigation trials and BMI control, despite the fact that reward was present in both cases (Fig. 4). In 
the former case, PDHs were near-flat and in the latter they showed a steady increase in activity. This finding points 
to factors different from a simple reward expectation signal as the main contributor to distance tuning. Likely, the 
steady increase in neuronal activity with target proximity reflected an increase in the level of animal engagement 
in the brain-control task. Indeed, when the monkeys actively controlled the wheel chair steering using their brain 
activity they probably attended more to the final target than when they were passively moved towards the end 
goal since in the latter condition the monkeys’ reward was guaranteed. Numerous previous studies offer possible 
contributing factors to the emergence of such a distance to reward tuning. For instance, cortical encoding of spa- 
tial attention, independent of the representation of motor goal and working memory, could have played a key role 
in our paradigm”*!. Furthermore, anticipatory buildup in cortical neuronal activity has been demonstrated by 
many researchers'*?!-%3 as potentially representing motor preparation (preparation of arm reaching in our case), 
spatial encoding”, and mirror-like representations**-*° where cortical activity reflects an observation of move- 
ment. All these potential factors, in principle, can be independently tested to quantify their individual contribu- 
tion to the distance to target tuning observed here. However such manipulations were beyond the scope of the 
present study and will be left for examination in future experiments. Moreover, we also intend to incorporate 
wireless multichannel, surface EMG recordings, to this setup to quantify whether any residual arm movements 
could contribute to these neuronal firing modulations. 

Irrespective of its origin, we learned that the distance to reward could be decoded offline using the corti- 
cal neuronal ensemble activity recorded in our experiments, yielding very reasonable and accurate predictions. 
Previously, we had decoded target location for an arm reaching task, where target location was also not part of the 
BMI control variables*’. The presence of such novel cortical representation suggests that, in the future, potentially 
new control parameters may be extracted from cortical neuronal ensembles, in addition to the variables nor- 
mally employed to control a BMI. Indeed, such emergent properties could add a new level of versatility to BMIs. 
Recently Marsh et al.* proposed that cortical representation of reward should be incorporated in BMI design to 
make it more autonomous. Our study adds to these results by describing the existence of neuronal signals which 
modulate not just the presence or absence of reward, but a continuous representation of reward location; a tuning 
of space that coexisted with the representation of chair velocity. In this context, future studies should explore the 
possibility that cortical neuronal ensembles engaged in navigation could trigger an incorporation of the navigated 
space (not necessarily related with reward location only) into novel cortical representations. 

Earlier we argued on a similar ground that prolonged BMI use leads to the assimilation of the artificial tool, 
e.g. robotic or virtual arms/legs, into the body representations found in the subject’s cortex**. Some evidence of 
such integration has been reported already for manually controlled wheelchairs**“*!. Our current observations 
of neuronal tuning to wheelchair kinematics and target distance after the transition to BMI control adds to these 
findings and suggests new directions for future scientific inquiries on how prolonged BMI operation affects cor- 
tical representation of the sense of self and the peri-personal space. 

Overall, the whole-body navigational BMI introduced here constitutes the natural evolution of our lab’s exper- 
imental approach of employing large-scale intracranial neuronal ensemble recordings as the main control signal 
source for neuroprosthetic devices®'®. Although we have seen a surge in the use of such BMIs in humans**>?, 
the majority of current clinical BMI applications are still dominated by noninvasive rather than intracranial BMIs. 
Part of the reason for this preference likely comes from the fact that intracranial BMIs have produced mixed clini- 
cal results, including an implant malfunction and failure, and motor outcomes that are not much superior to those 
obtained by non-invasive technology**”. A key issue accounting for these lackluster clinical results is related to 
the deterioration of neural recordings produced with Utah probes”*. Recently better results were reported for one 
human patient from whom ~200 motor cortical neurons were recorded for 13 weeks», but it remains to be seen 
how consistent these implants are. On the contrary, chronic implants in nonhuman primates based on flexible 
microwire arrays or cubes have been shown to yield good quality recordings for more than 5 years, even when 
hundreds of these metal filaments are implanted in the cortex’. In this context, microwire arrays maybe a better 
option than Utah probes for human chronic implants. Yet, in our judgment, more animal experimentation will be 
needed before reliable intracranial implants can be offered for humans’. 

In the wheelchair domain, numerous noninvasive EEG-based BMIs have been described. These applications 
employed motor imagery*® and P300 potentials*’ as the source of brain-derived motor commands. Although 
these systems perform with an acceptable (80%) success rate in tasks that involve predefined paths and target 
locations, and can be improved to cope with real environments”, they are clearly limited and cannot be consid- 
ered as the final solution for routine clinical use in the future. One approach to improve these systems is to apply 
shared control schemes where some commands are delegated to the robotic system*"*. With regard to our cur- 
rent study we plan to integrate navigation in complex environments in future experiments. 

The potential of intracranial BMIs for whole-body navigation, as demonstrated in our study, lies on the gen- 
eration of continuous kinematic control signals, which are needed for subjects to navigate through complex and 
unknown spaces. Since typical EEG based wheelchair design only supports discrete control, it is clearly insuf- 
ficient to generate continuous trajectories that can change at a moment's notice. Yet, because of their low risk, 
EEG based systems will probably remain the dominant clinical BMI approach for a while, and even expand to 
include new applications, like, EEG controlled exoskeletons. However, as intracranial recording systems improve 
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in efficiency and safety, they will likely become more attractive to the clinicians and patients, particularly those 
suffering from devastating levels of body paralysis, in the future. 

In conclusion, the current results promise much more than a mere demonstration that monkeys can con- 
trol whole body navigation. Indeed, the finding that cortical ensembles can adapt to a whole-body navigation 
task propels BMIs to a new dimension and creates innovative avenues for exploration of this approach’s clinical 
relevance in the future. In this context, the present results support our recent clinical observation that when 
paraplegic patients are subjected to intense BMI training, they not only became capable of regaining walking, 
using a robotic exoskeleton, but they also can exhibit signs of partial neurological recovery of sensorimotor and 
autonomic functions*’. Based on our experimental and clinical observations, we raise the hypothesis that BMIs 
can lead to partial neurological recovery or even augment brain function because their chronic and continuous 
use may trigger widespread cortical plasticity and the emergence of new cortical representations. As such, BMIs 
will likely have a profound clinical impact in the future. 


Materials and Methods 
Study Design. The objective of this study was to demonstrate whole-body navigation with control signals 
derived from neuronal ensemble recordings in multiple cortical areas. Notably, our design did not require the 
subjects to map overt movements to the navigational direction, which makes this paradigm applicable to the 
needs of severely paralyzed patients who cannot move their limbs but desire to restore whole-body mobility. 

Two adult rhesus macaques (Monkey K and M) were used for the study. All animal procedures were per- 
formed in accordance with the National Research Council's Guide for the Care and Use of Laboratory Animals 
and were approved by the Duke University Institutional Animal Care and Use Committee. The two monkeys 
were chronically implanted with arrays of microwires in multiple cortical areas of both hemispheres (M1, S1, and 
PMd). Our recently developed multi-channel wireless recording system was employed to sample from hundreds 
of neurons in sensorimotor cortex simultaneously’°. Both monkeys learned to navigate in a room while seated 
in a mobile robotic wheelchair using their cortical activity as the navigation control signal. Cortical ensemble 
recordings were converted to steering commands for the mobile robotic wheelchair based on linear decoding 
algorithms. 

Both monkeys successfully acquired the ability to steer the robotic wheelchair towards a grape reward. They 
achieved two dimensional navigation with multiple starting positions and orientations of the robot. 


Task Design. The monkeys were operantly conditioned to drive the mobile robotic wheelchair toward a food 
reward (grape). The rewards were delivered by an automated grape dispenser. Each experimental session con- 
sisted of about 150 trials during which the monkeys navigated to the grape from a starting location, reached for 
the grape and placed it in the mouth. The robotic wheelchair was then driven away from the grape dispenser to a 
new starting location. 

The first 30 trials were used to train BMI decoders. During these training trials, the monkeys had no control 
over the movements of the robotic wheelchair. Rather, the experimental control system navigated the wheel- 
chair along several predetermined routes toward the grape dispenser (Fig. $1). These training routes were chosen 
empirically. They represented distorted straight lines toward the grape dispenser. The distortions were achieved 
by adding moderate turns and backward movements. The data obtained during this passive navigation was used 
to train L2-norm Wiener Filters, which generated steering commands from a 1 s window divided into ten 100 ms 
bins to count spikes produced by each neuron in the ensemble. The filtering outputs were calculated as sums of 
the spike counts multiplied by the filter weights. 

Once the decoders were trained, we proceeded to BMI trials during which the robotic wheelchair was steered 
directly by the cortical signals. The monkeys would navigate themselves toward the grape, while the wheelchair 
returning to a starting location was the only part performed automatically by the program. (Obviously the mon- 
keys did not want to move away from the food location.) 

In the easiest version of the task, the two monkeys performed 1D navigation (8 sessions for Monkey K; 9 ses- 
sions for Monkey M). Here, the monkeys controlled only the forward and backward movements of the robotic 
wheelchair. The main, more challenging task required the monkeys to navigate in 2D. Their cortical activity 
was converted into both translational (backward or forward) and rotational (leftward or rightward) velocity 
components. 

The translational velocity was limited to — 0.28 to 0.28 m/s (negative values for backward movements and 
positive for forward), and rotational velocity was limited to — 46 to 46 degrees/s (negative for leftward, positive 
for rightward). If the decoded velocity commands exceeded the limit, the command sent to the robotic wheel- 
chair would be set to the limit value. In the 2D task, the robotic wheelchair could be initially placed at one of the 
3 starting locations, whose coordinates were (0.62, 0.50), (0.88, 0), and (0.62, —0.50) meter, where (0, 0) was the 
room center. 

The monkeys navigated in a 3.5-by—2.5 m experimental room. The actual drivable area was chosen to be 
3.1-by—2.4m, which assured safety. When the robotic wheelchair was at the drivable area boundary, the program 
would stop the robotic wheelchair if the decoded commands would have moved the robot through the boundary, 
and would only execute the decoded commands if the robot moved inside the drivable area. Additionally, when 
the robotic wheelchair got close to the grape dispenser (docking range), the program would take over the control 
and park the robot automatically to ensure the monkeys could comfortably obtain their rewards. 


Electrode Implantation. Monkey M was implanted with four multielectrode arrays. Within each array, 
Isonel coated, stainless steel polyimide insulated microelectrodes were grouped in two four-by-four, uniformly 
spaced grids each consisting of 16 microwire triplets of different lengths, for a total of 96 microelectrodes. 
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Although the implants were placed bilaterally in the primary motor (M1) and primary somatosensory cortices 
(S1) in both monkeys, in this study we recorded only from the right-hemispheric implants in Monkey K due to 
their superior neuronal yield. Each hemisphere received two arrays: one for the upper and one for the lower-limb 
representations (for a total of 384 microelectrodes, Fig. 1). In the current experiment we used 256 channels in 
Monkey M to record from neurons in the bilateral arm and leg areas of M1 and S1. Monkey K was implanted 
bilaterally with six multielectrode arrays containing 96 microwires each. Stainless steel microwires 30-50 um in 
diameter were grouped into bundles of three with a single leading conical microwire surrounded by two other 
microwires with cut angled tips’. The arrays were implanted in the arm and leg representation areas of M1 and 
S1 and also in the bilateral premotor cortices totaling 576 channels. For the current study we recorded from 128 
channels in Monkey K which were from bilateral M1 and premotor cortex and right side $1. Extracellular neu- 
ronal recordings were obtained using an in-house built wireless recording system described previously’®. 


BMI Navigation System Design. The BMI navigation system had three main components: (1) experiment 
control system, (2) wireless recording system, and (3) robotic wheelchair (Fig. $9). The experiment control system 
controlled the experimental sequence, performed BMI decoding of neuronal ensemble activity, and handled 
the video tracking of both wheelchair and monkey head pose. The wireless recording system recorded neuronal 
ensemble activity from the monkey brain and sent the neuronal data to the experiment control system. The 
robotic wheelchair accommodated the monkey chair and received driving commands wirelessly from the exper- 
iment control system. The three components communicated with each other in a local network. 


Experiment ControlSystem. The experiment control system controlled the flow of the experiment, which 
included controlling the experimental sequence, running BMI decoding algorithms that computed navigation 
commands for the robotic wheelchair, controlling the robotic wheelchair, and video tracking the wheelchair and 
the monkey head pose. 

The location and orientation of the robotic wheelchair was tracked using ASUS Xtion camera (640 x 480 
pixels, 30 frames/sec) mounted on the ceiling of the experiment room. The video tracking software was written in 
C+ +and built with OpenCV library. The software processed the video stream, and segmented the video frames 
to determine the robotic wheelchair location. The wheelchair orientation was determined based on the markers 
located at the front and the back of the robot. The robot position signals were smoothed using a Kalman filter. 
Similarly, we put color markers on the cover of the monkey headcap to track its location and orientation the same 
way as we tracked the wheelchair. 


Statistical Analysis. We ran Monkey K for 21 sessions and Monkey M for 28 sessions, and sometimes two 
sessions in a day. Two sessions of Monkey K and five sessions of Monkey M were excluded from the analysis 
because (1) the sessions had less than 10 BMI navigation trials (3 sessions), (2) technical issues occurred during 
recording (1 session), (3) a decoder was ill-trained and biased toward negative velocity (1 session), or (4) commu- 
nication between the experiment control system and the Raspberry Pi on the robot failed frequently (2 sessions). 
On average, each session yielded 49.0 + 32.0 and 44.4 + 9.3 BMI navigation trials by Monkey K and M. In the end, 
19 sessions from Monkey K and 23 sessions from Monkey M entered the analysis below. 


Tuning Diagram. To characterize tuning diagrams for each neural unit at different time lags (e.g., Fig. 2A), 
we first aligned the controlling commands (i.e., translational and rotational velocity commands) of the robotic 
wheelchair to the neural activities (spike counts of 100 ms time bin). Next, the spike counts were normalized by 
subtracting the mean and dividing by the standard deviation of the spike counts of this unit. The range of the 
controlling commands were both divided into 18 bins, which formed the two axes of the tuning diagram (18 x 18 
bins). Then, we sorted the normalized spike counts into the diagram based on the corresponding controlling 
commands, and averaged the normalized spike counts that fell into the same bin. Lastly, missing values of the 
tuning diagram were filled in and smoothed by the mean of neighbors in both space and time. To quantify how 
similar two tuning diagrams were, we calculated Pearson correlation between the two diagrams. 


Neuronal Modulation to Distance to Target. The firing rates for each session were first binned into 
10 ms-bins and normalized against the session averages. The maximum range of distance the wheelchair traveled 
to was divided into 20 bins. For each session, the normalized firing rates of all neurons corresponding to when 
the monkey was within a particular distance bin during each regime (training vs. brain-control) were averaged 
to obtain the session average. The resulting session averages were then averaged to obtain the total average. These 
were plotted in Fig. 4’s raster plots. 

To obtain the histograms from the total session averages, we took the average of the absolute values of all neu- 
ron’s normalized firing rates at each distance bin. The reach-dependent distance (the bars colored red in Fig. 4) 
was a conservative estimate of when a monkey may reach the grape dispenser. 
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OPEN Place Cell-Like Activity in the 
Primary Sensorimotor and 
-Premotor Cortex During Monkey 
<antumes  Whole-Body Navigation 


po ae cee A. Yin12, P. H. Tseng?3, S. Rajangam?3, M. A. Lebedev?? & M.A. L. Nicolelist2:345 


Primary motor (M1), primary somatosensory (S1) and dorsal premotor (PMd) cortical areas of rhesus 
monkeys previously have been associated only with sensorimotor control of limb movements. Here we 
show that a significant number of neurons in these areas also represent body position and orientation 
in space. Two rhesus monkeys (K and M) used a wheelchair controlled by a brain-machine interface 
(BMI) to navigate in a room. During this whole-body navigation, the discharge rates of M1, S1, and PMd 
neurons correlated with the two-dimensional (2D) room position and the direction of the wheelchair 
and the monkey head. This place cell-like activity was observed in both monkeys, with 44.6% and 33.3% 
of neurons encoding room position in monkeys K and M, respectively, and the overlapping populations 
of 41.0% and 16.0% neurons encoding head direction. These observations suggest that primary 
sensorimotor and premotor cortical areas in primates are likely involved in allocentrically representing 
body position in space during whole-body navigation, which is an unexpected finding given the classical 

: hierarchical model of cortical processing that attributes functional specialization for spatial processing 

: to the hippocampal formation. 


Little is known about the role of sensorimotor cortical areas, such as the primary motor (M1) and somatosensory 
(S1) cortices, in representing parameters of whole-body navigation. To date, neurophysiological mechanisms 
enabling whole-body navigation have been most extensively studied in the rat hippocampus'“, where “place 
: cells” representing spatial locations and navigation parameters were originally reported by O’Keefe and his col- 
: leagues!”. Much less neurophysiological research has been conducted in nonhuman primates, employing virtual 
navigation tasks’ and whole-body motion®””. In humans, neural correlates of navigation have been investigated 
using electroencephalography (EEG)'?’, neuroimaging’*"*, and intracranial recordings'*"!’. Our recent studies 
in rhesus monkeys have shown that cortical motor (M1 and PMd) and somatosensory (S1) areas contribute to the 
encoding of body position during whole-body navigation”. This is surprising because, according to the classical 
hierarchical model of cortical processing’”’, space-coding should be restricted to the interactions between the 
hippocampus and association areas of the cortex. 
To clarify the role of different brain areas in controlling whole-body navigation, we®'* and others** have 
recently started using a new experimental paradigm, where monkeys navigate in a room while seated in a 
motorized wheelchair. In these settings, monkeys can learn to control their wheelchair navigation®’”* using a 
: brain-machine interface (BMI). In our implementation of the BMI for driving’, the velocity commands to the 
: wheelchair were generated from the linearly combined activity of cortical neuronal ensembles recorded in the 
> primary motor (M1), primary somatosensory (S1), and the dorsal premotor cortex (PMd). In the same study, 
: we reported that a large fraction of cortical neurons in M1, $1, and PMd were tuned to the distance between the 
: monkey and the target of navigation. This observation led us to hypothesize that monkey cortical neurons in these 

areas could also encode spatial location, which bears similarity to the encoding of space by the rodent place cell. 

Here, we report the results of testing this hypothesis. Our findings reveal that, in addition to representing arm 
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reaching and wheelchair kinematics in egocentric coordinates, M1, S1 and PMd neurons allocentrically represent 
the monkey's body location, as well as head and body orientation. Altogether, these findings suggest that hier- 
archically low cortical areas contribute to the representation of allocentric space during whole-body navigation. 


Results 

Two monkeys (K and M) were employed in our experiments. During each recording session, a monkey sat in 
a motorized wheelchair and navigated from one out of three possible starting locations in a room towards the 
location of a grape dispenser (Fig. 1A). This whole-body navigation was performed under BMI control, where a 
linear decoding algorithm transformed cortical ensemble activity into translational and rotational velocity com- 
ponents responsible for the wheelchair movements’. The wheelchair passed through different room locations, 
which allowed us to construct position tuning maps that described how firing rates of individual M1, $1, and 
PMd neurons depended on wheelchair position (Fig. 1B-G). While most of the room was covered by the naviga- 
tion trajectories, the coverage was not uniform for all locations, with the edges of the room being less visited. The 
distribution of the trajectories was also dependent on the wheelchair starting location — those starting at one side 
of the room tended to stay on that side. For analysis purposes, the region of the room where the grape dispenser 
was located was defined as the “front of the room” (Fig. 1A), whereas the “back of the room” corresponded to the 
region from which the wheelchair started to move. The “left” and “right parts of the room” corresponded to the 
monkey’s view when it faced the dispenser, the “front of the roomy. We analyzed data from nine daily sessions in 
monkey K and 23 sessions in monkey M. Spike sorting was conducted on day one; afterwards the sorting param- 
eters were adjusted when changes in neuronal waveforms were noticed. We analyzed activity patterns of 116 
neurons in monkey K (27 in S1, 64 in M1, and 25 in PMd; see Supplementary Materials for selection criteria) and 
124 neurons in monkey M (59 in S1 and 65 in M1). 


Individual neurons are tuned to both spatial location and orientation. Figure 1B-G shows that, 
in both monkeys, individual M1, S1, and PMd neurons modulated their firing rates according to the wheelchair 
spatial position. The position tuning of these neurons reflected not only the distance, r, from the wheelchair to 
the grape dispenser, but also whether the wheelchair was in the left or right part of the room. For example, the 
M1 neurons (from both monkeys) shown in Fig. 1B,E increased their firing rate when the wheelchair entered the 
right part of the room. Meanwhile, the PMd neuron shown in Fig. 1C (recorded in monkey K) showed an oppo- 
site pattern of position tuning: its firing rate decreased when the wheelchair entered the right part of the room and 
increased when it entered the left part. Figure 1D,F depicts an M1 neuron from monkey K and an S1 neuron from 
monkey M that, despite being very weakly tuned to the left-right dimension, were clearly tuned to the distance 
from the wheelchair to the grape dispenser: the firing rate increased in both neurons with decreasing distance. 
Figure 1G depicts an M1 neuron from monkey M whose firing rate decreased as the wheelchair approached the 
grape dispenser. In many neurons (see Fig. 1D-G), we observed a sharp change in firing rate that occurred when 
the wheelchair entered the “reach zone” (marked by black semi-circles in Fig. 1), from which the monkeys could 
reach for the food. Since monkey K was a smaller monkey, its reach zone was also smaller compared to the zone 
for monkey M. 

In addition to the reach zone, Fig. 1 also shows the “docking zone” (marked by red circles). Inside this region, 
an autopilot program took over and maneuvered the wheelchair to dock it next to the dispenser to facilitate the 
monkey’s access to the grapes. For monkey M, the reach zone and the docking zone mostly coincided, since video 
analysis showed that this monkey began reaching while the wheelchair was being docked. Conversely, monkey K 
initiated reaching only when the wheelchair traveled close enough to the grape dispenser. This difference in the 
onset of reaching in the two monkeys was clearly visible in the neuronal patterns: a transition to reach-related 
activity occurred in monkey K only within the small reach zone (Fig. 1D), whereas in monkey M neuronal activ- 
ity sharply changed when the wheelchair entered the docking zone (Fig. 1E-G). Overall, Fig. 1 shows that S1, M1 
and PMd neurons in both monkeys were tuned to the 2D room position, including tuning in both left-right and 
back-front dimensions, while exhibiting, in addition, the well-known typical firing modulations related to arm 
reaching and hand grasping movements that have been classically associated with these cortical areas. 

Further analysis revealed that, in addition to representing room position, the firing rates of M1, S1, 
and PMd neurons were affected in varying degrees by the direction relative to room landmarks in which 
the monkeys turned their heads and the rotations of the wheelchair. We named this type of neuronal tuning 
“orientation-related”. To illustrate orientation tuning, Fig. 2A-D compares the instances when the monkey’s head 
pointed to the right of the grape dispenser (45 < a < 135, Fig. 2A,C) to the instances when the monkey’s head 
pointed to the left of the dispenser (— 135 < a < —45, Fig. 2B,D). Two neurons from monkey K are illustrated: a 
PMd neuron exhibited a strong preference to the right part of the room, irrespective of where the monkey’s head 
pointed (Fig. 2A,B), while an M1 neuron displayed firing patterns that were clearly different in these two cases; 
when the monkey's head pointed to the right of the dispenser (Fig. 2C) this neurons firing rate was much lower 
than when the monkey’s head pointed to the left (Fig. 2D). In general, we observed that M1, $1, and PMd neurons 
represented both room location and monkey head orientation, with some neurons being more location-tuned 
and others more orientation-tuned. 

To estimate the proportion of neurons that show significant tuning to spatial location and body/head orien- 
tation, we conducted a mutual-information analysis!®*475 that evaluated the correspondence between neuronal 
rates and individual task covariates (spatial location, monkey head angle a, and wheelchair angle (; Fig. S1 and 
Table S1). In monkey K, significant mutual information (permutation test with FDR correction, p < 0.05) to at 
least one of these three covariates was found for 80.4 + 12.9% neurons (mean + std across sessions) (73.3 + 21.1% 
in S1, 84.7 + 11.6% in M1, and 76.9 + 12.3% in PMd). In monkey M, 47.5 + 18.7% neurons were tuned to at least 
one parameter (47.2 + 18.4% in S1 and 47.9 + 19.5% in M1). Tuning to room position was found for 64.0 + 18.8% 
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Figure 1. Wheelchair navigation setup and examples of spatially-selective neurons. (A) View from the top of 
the experimental room, where the monkey navigated in a motorized wheelchair under BMI control from one of 
three possible starting locations (shown as green circles) to the grape dispenser (marked by a ‘+’). The four 
walls of the experiment room are labeled as “front”, “back” “right” and “left”, which correspond to the monkey 
facing the grape dispenser and navigating from the back to the front of the room. Red semicircle labels the 
docking zone upon which an auto-pilot took over the wheelchair control. Black semicircle labels the reach zone 
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within which monkeys initiated reach movements for the grape. The position of the monkey’s body and 


orientation of the head (gray ellipse) and trunk (tan ellipse) were evaluated using the following coordinates: - 
the angle of the wheelchair’s location with respect to the dispenser; r — the distance from the wheelchair to the 
dispenser; a- the angle of the dispenser location with respect to the monkey’s head direction; and /3- the angle 


of the dispenser location with respect to the wheelchair’s direction. (B-D) Spatial tuning diagrams in 


representative neurons from monkey K and monkey M (E-G), axis labels as in (A). Color represents trial- 
average z-scored firing rates for different room locations. Cortical areas, where the neurons were recorded, are 


indicated. 
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Figure 2. Neuronal tuning to room location and head orientation. (A-D) Modulation of spatial tuning 
patterns by head orientation in two neurons from monkey K. Conventions as in Fig. 1B-G. The color plots 
correspond (see key on top) to the right from the grape dispenser (a from 45 to 135 degrees; A, C) or to the left 
from the dispenser (a from —45 to —135 degrees; B, D). The spatial tuning pattern of one PMd neuron did not 
substantially change with the head orientation (A,B), whereas the pattern of another M1 neuron dramatically 
changed (C,D). (E-I) Scatterplots of tuning depth (TD) to room location versus TD to orientation (« and (3) for 
monkey K’s (E-G) or monkey M’s (H-I) neurons with significant mutual information to either parameter from 
all sessions, across cortical areas: $1 (E, H), M1 (F, I) and PMd (G). The diagonal lines show where space-TD 
equals to orientation-TD. The dashed ellipses below and above the diagonal lines illustrate clusters of highly- 
tuned position-preferring and orientation-preferring neurons, respectively. 
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(63.0 + 20.6% in S1, 62.8 + 20.3% in M1, and 68.0 + 15.7% in PMd) and 36.5+ 16.1% (35.4+ 14.7% in S1, and 
37.4+ 17.7% in M1) in monkey K and monkey M respectively. 

To quantify the dependence of the firing rate of individual neurons on position and rotational covariates, 
we pulled out the neurons with significant mutual information to position or orientation parameters. Next, we 
applied generalized additive models”®”” (GAM) to these neurons. This analysis was complicated by the fact that 
orientation and position parameters were mildly correlated (Pearson correlation coefficient <0.3 for all sessions). 
To cope with this issue, we included all second-order interaction effects between covariates. We defined the neu- 
ronal tuning depth (TD) of a neuron to a covariate or a group of covariates as the fraction of that neuron’s firing 
rate deviance uniquely explainable by that covariate or group of covariates (see Methods). 

To assess the composition and properties of the neuronal populations tuned to room position and wheel- 
chair and head orientation, we grouped the covariates of our GAM into position-related (x and y) and 
orientation-related (a and 8). Next, for each neuron entered into the GAM analysis, we calculated its position and 
orientation neuronal TD, respectively, and plotted position TD against the orientation TD values (Fig. 2E-I). In 
these scatter-plots, each point represents an assessed neuron with significant TD (F-test, p < 0.005) to at least one 
parameter. The scatter-plots showed that many M1, $1, and PMd, neurons were tuned to both room position and 
orientation parameters. Of the neurons with significant mutual information to position or orientation parameters 
that were used in the GAM analysis, 30.4 + 17.6% (mean + std across sessions) in monkey K (S1, M1, and PMd) 
and 25.7 + 18.5% in monkey M (S1 and M1) showed significant tuning to both parameters (see Table 2-3). 
While representing both position and orientation, neurons could still be classified as position-preferring versus 
orientation-preferring (above and below the diagonal lines, respectively, in Fig. 2E-I). Additionally, for each 
monkey, we defined highly tuned neurons as those with preferred TD (for position or orientation) exceeding 
the median TD for the neuronal sample that entered the GAM analysis in the corresponding recording sessions. 
The proportions of highly tuned neurons and neurons with preferences for either position or orientation tun- 
ing varied according to cortical area (S1, M1, or PMd) and monkey (K or M). In both monkeys, the distribu- 
tion of TDs was positively skewed because of the presence of highly-tuned neurons (Supplementary Fig. $3). In 
monkey K, the TDs for highly tuned neurons, preferring either location or orientation, were markedly stronger 
(Kruskal-Wallis, p < 0.001; «=0.05 for Tukey’s multiple comparison; on TDs pooled over all sessions) for PMd 
neurons (0.011 + 0.020; session pooled median + interquartile range) than for M1 neurons (0.005 + 0.003) and 
S1 neurons (0.006 + 0.004). The GAM showed that most neurons were position preferring in PMd, where they 
constituted 81.9 + 6.5% (mean + std) of all neurons and 91.5 + 5.0% of highly tuned neurons. In M1, 32.1+19.0% 
of all neurons and 44.8 + 13.1% of highly tuned neurons were position-preferring. In S1, 53.1 + 16.8% of all neu- 
rons and 71.9 + 16.6% of highly tuned neurons were position-preferring. The percentage of all neurons that were 
position-preferring in PMd was significantly higher than in M1 (Friedman's test, p < 0.001; a=0.05 for Tukey’s 
multiple comparison) but not than in S1. The percentages of highly-tuned neurons that were position-preferring 
in both PMd and S1 were significantly higher than in M1 (Friedman’s test, p < 0.001; a=0.05 for Tukey’s multi- 
ple comparison). Although M1 had fewer position-preferring neurons, a clear cluster of neurons with high TD 
for position was found in this area (marked by an ellipse in Fig. 2F). In monkey M, preferred TDs were lower 
compared to monkey K (p < 0.005, Wilcoxon rank-sum test, neurons pooled over sessions; see also Fig. $3), and 
position-preferring and orientation-preferring neurons were approximately equally represented in both M1 and 
S1 (Friedman's test; See also Table S3). 


Population tuning characteristics. To visualize the position-tuning properties for the populations 
of position-preferring neurons, we plotted the session average heatmaps that represented ensemble-average 
place fields for the position-preferring neurons; only the room locations outside the docking zone are shown 
(Fig. 3A,B). For each neuron, a place field was defined as the room locations where the neuron’s tuning function 
fitted by GAM exceeded the median value over all room locations. For both monkeys, the heatmaps revealed 
non-uniform distributions of neuronal place fields (Kolmogorov-Smirnov test, p < 0.01), with gradients of neu- 
ronal activity in back-front and left-right dimensions. In monkey K neuronal place fields were concentrated in 
the front-right part of the room. To visualize orientation tuning, we plotted the average histogram of the preferred 
a (monkey head angle) and 6 (wheelchair angle) directions for the orientation-preferring cells over all sessions 
(Fig. 3C-F). A diversity of preferred directions was found (Kolmogorov-Smirnov test, p < 0.01) but their distri- 
butions were different, with peaks around 135 degrees (i.e. facing away from the grape dispenser) for both a and 
8 in monkey K (Fig. 3C,E), and peaks around 0 and 180 degrees (ie. facing either toward or away from the grape 
dispenser) for « and 8 in monkey M, respectively (Fig. 3D,F). 


Spatial tuning properties across different days. By recording from the same sample of neurons over 
several daily sessions, we observed that the positional and orientation tuning properties of individual cells often 
remained the same over multiple days. Figure 4A illustrates that the position tuning of monkey K’s PMd neuron, 
the same as the one shown in Fig. 1C, remained stable over nine recording sessions. This neuron consistently 
increased its firing rates when the wheelchair entered the left side of the room (also see Fig. 4B for modulations in 
the firing rate of the same neuron along individual navigation trajectories). Figure 4C illustrates the position tun- 
ing of monkey K’s M1 neuron shown in Fig. 1B, which remained stable over six recording sessions. This neuron 
increased its firing rates when the wheelchair was on the right side of the room. Figure 4D illustrates the nine-day 
stability of neuronal spatial patterns in a highly-tuned position-preferring PMd neuron from monkey K, which 
increased its discharge rates as the wheelchair entered the right side of the room but then decreased firing close to 
the right wall. Figure 4E illustrates the nine-session stability of neuronal place-fields patterns in a S1 neuron from 
monkey M, which increased its discharge rates as the wheelchair entered the front of the room. 

To estimate the proportion of cells with stable spatial tuning, we first selected neurons from each monkey 
that were highly tuned for at least 70% of all recording sessions. This selection assured that we analyzed only very 


SCIENTIFIC REPORTS | (2018) 8:9184 | DOI:10.1038/s41598-018-27472-4 5 


www.nature.com/scientificreports/ 


A Monkey K B Monkey M 


high-activity probability 


270° 270° 


Figure 3. Place fields and preferred-directions for the neuronal ensembles. (A,B) Heatmaps of ensemble- 
average place fields for monkey K (A) and monkey M (B); plots represent averages across all sessions. Only 
position-preferring neurons were included. Each neuron’s place field was defined as the room locations where 
the neuron’s tuning function (as fitted by GAM) exceeded the median value over all room locations. A 
histogram of place-fields was constructed from all position-preferring cells’ place-fields and normalized by the 
number of position-preferring cells in each session. These session histograms were then averaged to produce the 
final heatmap. (C,D) Circular histograms of preferred a directions averaged over sessions for monkey K (C) and 
monkey M (D). For each session, the preferred a directions of all orientation-preferring neurons were extracted 
from their a-tuning curves (as fitted by GAM). The preferred directions were then represented as circular 
histograms. Radial direction represents proportion of neurons. (E,F) Circular histograms for 3. Conventions as 
in (C,D). 
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Figure 4. Individual neurons with consistent position tuning across multiple sessions. (A) The position-tuning 
diagram of monkey K’s PMd neuron shown in Fig. 1C for all nine experimental sessions. Session ordered from 
left to right, then top to bottom. The first session displayed corresponds to Fig. 1C. (B) Sample brain-controlled 
navigation trajectories for the same session as the data shown in Fig. 1C. Colors represent normalized firing 
rates along the trajectories. Black “** mark the location at the end of each trial, and black ‘o mark the starting 
locations of each trial. (C) The position-tuning diagram of monkey K’s M1 neuron shown in Fig. 1B, for the first 
six experimental sessions. Session number increases from left to right. The first session is the same as shown in 
Fig. 1B. (D) Position tuning diagrams of monkey K’s PMd neuron for all nine experimental sessions. Session 
ordered from left to right, then top to bottom. (E) Position tuning diagrams of monkey M’s S1 neuron shown 

in Fig. 1F for nine consecutive experimental sessions, just before the session shown in Fig. 1F. Session ordered 
from left to right, then top to bottom. 


stable neurons (in terms of the presence of position tuning) and disregarded the less stable units. In the subse- 
quent analyses, all of the selected neurons were considered even if some of them did not have a significant MI in 
a given session. About 35.3% in monkey K (33.3% in S1, 34.4% in M1, and 40.0% in PMd) and 12.1% in monkey 
M (15.3% in S1 and 9.2% in M1) fulfilled this criterion. Using this subset of cells, we then quantified the similarity 


SCIENTIFIC REPORTS | (2018) 8:9184 | DOI:10.1038/s41598-018-27472-4 7 


www.nature.com/scientificreports/ 


between the neurons’ tuning patterns from pairs of sessions as their correlation coefficients (i.e. between-session 
consistency). Correlation coefficient values were computed similarly between halves of the same session (i.e. 
within-session consistency, see Methods). All correlation values were significantly different from zero (permu- 
tation test, p< 0.001) and there was no significant difference between within- and between-session correlation 
values (Wilcoxon rank-sum test). Thus, the across-session variability of neuronal patterns was the same as the 
within-session variability, indicating that neuronal spatial tuning properties stayed unchanged across multiple 
days. 


Decoding of position. The position tuning properties of the recorded neurons allowed us to conduct offline 
decoding of the wheelchair’s position using neural network decoders. All these decoders had one hidden-layer 
and used spiking activities from either all neurons with significant MI (all), position-preferring neurons 
(position-pref), or orientation-preferring neurons (orientation-pref). The decoder performance was evaluated by 
calculating the mean prediction error (MPE) in meters, defined as the average distance between the decoded and 
actual locations pooled over the experiment room (Fig. 5A). In monkey K, all and position-pref decoders per- 
formed significantly better than chance for all sessions, orientation-pref decoders performed better than chance 
for 7 out of 9 sessions (permutation test, p< 0.05). The all and position-pref decoders significantly outperformed 
orientation-pref decoders (Kruskal-Wallis, p <0.01; a=0.05 for Tukey’s multiple comparison). Figure 5B-D 
shows MPE at different room positions for a representative session of monkey K. The all and position-pref decod- 
ers predicted location with low errors over large areas of the experimental space (Fig. 5B,C). The orientation-pref 
decoders performed well near the center of the room, but poorly around the left and right boundaries (Fig. 5D). 
In monkey M, all and orientation-pref decoders performed significantly better than chance for 11 out of 23 ses- 
sions, position-pref decoders performed significantly better than chance for 10 out of 23 sessions (permutation 
test, p < 0.05). Notably, the average trial durations in sessions with significant decoder performance were signifi- 
cantly shorter than those in sessions with chance-level decoder performance (p < 0.01, Wilcoxon rank-sum). All 
three types of decoders had similar performances, with the position-pref decoders yielding marginally better pre- 
dictions (Fig. 5A). Monkey M’s decoders all predicted position well near the room center but poorly near room 
boundaries (Fig. 5E-G). Better decoding performance in monkey K compared to monkey M is not surprising, 
since monkey K’s neurons generally had higher TD to position than monkey M. Thus, a higher amount of posi- 
tion information translated into higher decoding performance. 

Orientation parameters (« and 8) were decoded similarly as well using all, position-pref, and orientation-pref 
decoders. The decoding performances measured in Pearson's product-moment correlation coefficient (cc) are 
shown in Figure $4. In monkey K, prediction of both orientation parameters was significantly better than chance 
for all of the sessions for all types of decoders used. The prediction of a by all and orientation-pref decoders 
were significantly better than that by position-pref decoders (Kruska- Wallis, p < 0.05 and p < 0.01, respectively; 
a= 0.05 for Tukey’s multiple comparison). In monkey M, prediction of a was significantly better than chance 
for all sessions with all and position-pref decoders, and for 20 out of 23 sessions with orientation-pref decoders. 
Prediction of « by all decoders were significantly better than that by orientation-pref decoders (Kruskal-Wallis, 
p <0.01; a=0.05 for Tukey’s multiple comparison). Prediction of 8 was significantly better than chance for 
all sessions with all and position-pref decoders, and for 22 out of 23 sessions with orientation-pref decoders. 
Prediction of 8 by all decoders were significantly better than that by orientation-pref decoders (Kruskal-Wallis, 
p <0.01; a=0.001 for Tukey’s multiple comparison). 


Decoding of arm reach state. As the wheelchair entered the reach zone, monkeys initiated reach move- 
ments for the grape. Population perievent time histograms aligned on the wheelchair entering the reach zone 
clearly showed reaching-related neuronal activities (Fig. S5). We classified the presence of reaching movements 
from these patterns of neural activities using decision-tree ensemble classifiers. Figure S5 shows the classifi- 
cation results in both monkeys as heatmaps of reach-state probability in different parts of the room for rep- 
resentative sessions. Classification was performed using different populations of neurons: all neurons with 
significant MI (Fig. S5C,F), subpopulations of position-preferring neurons (Fig. S5D,G) or subpopulations 
of orientation-preferring neurons (Fig. S5E,H). The classifier correctly recognized the reaching state in the 
smaller reach zone for monkey K (Fig. SSC-E) and the larger zone for monkey M (Fig. S5F-H). As the num- 
ber of units with significant MI, position- and orientation-preferring units varied across sessions, we plotted in 
Supplementary Fig. $51 the accuracy of classifiers as a function of the ensemble size. The accuracy was quantified 
by Cohen's «*, a metric that is appropriate for measuring classification accuracy on unbalanced datasets. All 
of monkey K’s classifiers and approximately half of monkey M’s classifiers, using all units with significant MI, 
achieved good accuracy (0.61 < « < 0.80, marked by the gray band) on the Landis and Koch scale”*. In both mon- 
keys, the classifiers’ performance increased with the ensemble size. This dependence was significantly different 
for the neuronal populations in monkey M (nonparametric ANCOVA, p < 0.001), but not in monkey K (nonpar- 
ametric ANCOVA)”. For both monkeys, after taking into account the sample size, the classification accuracy did 
not significantly differ by subpopulation type (nonparametric ANCOVA). 


Discussion 

In the present study, two rhesus monkeys employed a BMI to control the movements of a motorized wheelchair 
in order to navigate to a fixed target location in a room. This experimental setting allowed us to study cortical 
representation of the animals’ body position and orientation. We observed that a significant fraction of $1, M1, 
and PMd neurons exhibited spatial tuning, with both position and orientation encoded by neuronal discharges. 
These neuronal patterns resembled two classes of neurons previously reported in the hippocampus: place cells'~4 
and head-direction cells***!. Additionally, the spatial tuning properties of cortical neurons remained stable over 
several days, which is also similar to the place-fields observed in the hippocampal place-cells”*”. Since we did not 
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Figure 5. Decoding of room location from neuronal ensemble activity. (A) Box plots showing the distribution 
of decoding accuracy and chance decoding accuracy (left and right boxes within each group, respectively) 
measured by the mean prediction error (MPE) in meters, in monkey K and monkey M using different 
population of neurons. Only sessions with decoding accuracy significantly above chance are shown. The 
distributions of chance-level decoding accuracy are constructed from the lower-end of the 95% confidence 
interval of the selected sessions’ chance-level error. (**) indicates significant p-value < 0.01 for post-hoc Tukey’s 
multiple comparison following Kruskal-Wallis test. (B-G) Color plots of MPE as a function of room location 

in one representative session each for monkey K (B-D) and monkey M (E-G) - from using all neurons (B,E), 
position-preferring neurons (C,F), and orientation-preferring neurons (D,G). 


record from the hippocampus, it remains to be examined to what extent space-representing neuronal activity that 
we observed in navigating monkeys would match the hippocampal activity during the same task. We leave this 
question for further studies. 

There are several potential explanations that could account for our findings, different from cortical encoding 
of body position and orientation per se. For example, neurons in the sensorimotor and premotor cortical areas 
have been shown to encode rewards**", so it is possible that expectation of reward could explain neuronal tuning 
to the distance from the monkey to the grape dispenser? and the front-back position-selectivity. Yet, position 
selectivity in the left-right dimensions is inconsistent with this hypothesis. Reward expectation depends on the 
distance from the monkey to the reward, but not on the left-right dimension. Even if reward expectation affected 
the neuronal rates in our experiments, a similar result has been reported for monkey hippocampus”. In this latter 
study, the authors concluded that “the primate hippocampus contains a representation of the reward associations 
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of places”. Our findings may be also interpreted in a similar way: activity of M1, S1, and PMd neurons represented 
the 2D location of the monkey’s body with respect to reward. 

Motor preparatory activity” is another possible alternative explanation for our findings. One could suppose 
that monkeys were preparing arm reaching movements in different directions, while traveling in the wheelchair, 
for example leftward reaching when the wheelchair was close to the right wall of the room and rightward reach- 
ing when the wheelchair was near the left wall. The flaw with this potential explanation is that our monkeys 
never performed these “eventual” arm reaching movements in different directions. Instead, since these animals 
were over trained, in every session we noticed that only after the wheelchair was docked and the monkeys faced 
the grape dispenser did they produce a comfortable and stereotyped arm reach. Invariably, these arm reaches 
occurred only in the forward direction, relative to the monkeys’ body. As such, reach-related activity could not 
account for the tuning to position. In monkey K, position tuning in the left-right direction was especially strong. 
Yet, this monkey always initiated an arm reach after the wheelchair was securely docked and the movement 
was always in the forward direction. Conversely, monkey M often initiated arm reaching movements before the 
wheelchair was docked, but its neurons were poorly tuned in the left-right dimension during the navigation. 
Based on these considerations, we conclude that motor activity related to arm reaching movements offers a very 
unlikely explanation for the neuronal position encoding observed here. 

A more exotic explanation would involve a “virtual arm” that extends from the monkey to the food while the 
monkey navigates. This explanation would resemble Rizzolatti’s premotor theory of attention*” where a move- 
ment in a certain direction is prepared but never executed. This hypothesis, however, does not explain why a 
neuron remains tuned to the same spatial location when the chair is rotated. 

Another hypothesis is that the position-tuning patterns observed were related to planning and executing 
a whole-body trajectory in space. This explanation does not contradict the characterization of our findings in 
terms of place cells; rather, it could provide an explanation for the function of place-cell tuning in M1, PM, and 
S1. The explanation is also consistent with previous theories, since it has been suggested that allocentric spatial 
information from the hippocampus-entorhinal circuits is projected to the parietal cortex, where it is transformed 
into body-centered coordinates and then relayed to the motor and premotor cortices for execution”. Yet, if this 
were true, a prevalence of egocentric encoding would be expected for M1, $1 and PMd, which we did not observe 
in our experiments. 

In the future, it will be interesting to test whether the cortical place fields reported here rotate, like observed 
for hippocampal place-fields, following rotations of environmental cues**’. While this possibility has yet to be 
thoroughly tested, we have recently reported that place-fields of M1 and PMd neurons also change when the 
objects in the room are moved"*. In that study, one monkey (the Passenger) was carried by a wheelchair while a 
second monkey (the Observer) sat in a corner of the room observing the action. A grape dispenser was placed 
in an adjacent corner. While the Passenger navigated toward the grape dispenser along a convoluted trajectory, 
its M1 and PMd neurons exhibited positional tuning that were like the tuning reported here, but they reflected 
the Passenger’s position with respect to the two other salient objects in the room: the grape dispenser and the 
Observer monkey. In control sessions where the location of the Observer monkey was swapped with that of the 
grape-dispenser, the position-tuning patterns of M1 and PMd also flipped accordingly. Although more experi- 
ments would need to be done to examine this effect in more detail, these results confirm that the place fields of 
M1 and PMd are defined by the arrangement of objects in the monkey’s environment. 

Thus, our findings show that $1, M1, and PMd generate an allocentric representation of space, very much 
like hippocampal place cells. Additionally, our results are consistent with several studies conducted in differ- 
ent species showing allocentric spatial tuning of neurons outside the hippocampus. In the ferret primary audi- 
tory cortex, populations of neurons tuned to both allocentric and egocentric spatial location of sound were 
found“. The tuning properties were stable through several sessions. In the rat primary visual cortex"', subsets 
of neurons were found to be predictive of the upcoming visual stimulus in a position dependent manner. The 
position-dependency properties were found to be stable and the authors suggested that the visual cortex forms 
an internal representation of a visual scene based on spatial location, mediated by top-down projections from 
the hippocampus via the anterior cingulate cortex (ACC). In the mouse retrosplenial cortex (RSC)”, a subpop- 
ulation of neurons were found to express a sparse, orthogonal and continuous representation of a linear envi- 
ronment in a fashion highly similar to CA1 place-cells. The place-fields of these neurons were stable through 
time and could remap upon altering of spatial context. Place-cells in RSC were found to be most prevalent in 
the CA1 recipient layers, making the inheritance of spatial-information from the hippocampus a likely expla- 
nation. In the rat claustrum*’, a subcortical structure that has extensive direct and indirect connections with the 
hippocampal-entorhinal structures, populations of neurons were found to exhibit activities similar to place-, 
boundary-, and object-cells in the hippocampal-entorhinal complex. The place-cells were found to have stable 
place-fields but did not remap according to distal cues or proximal objects. In rat lateral septum’, neurons were 
observed to represent context-specific location-selective firing. The spatial-selective activities were stable across 
days and remapped to different areas. The authors attributed their observations to direct projections from the hip- 
pocampus. All these reported areas have direct connections with the medial temporal hippocampal-enthorhinal 
complex. 

Thus our results suggest that encoding of body position in space is much more distributed in the brain than 
previously thought. In this context, our findings expand considerably the extent of cortical circuits known to be 
involved in creating an animal's spatial localization to include the premotor, and the primary motor and soma- 
tosensory areas, which are considered as hierarchically low processing cortical regions because they have no 
direct connectivity with hippocampal structures. 
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Materials and Methods 

All animal procedures were performed in accordance with the National Research Council’s Guide for the Care 
and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use 
Committee. 


Study Design. The objective of this study was to examine the cortical representation during whole-body 
navigation. The experiments performed were described in detail in’, where we demonstrated whole-body naviga- 
tion with control signals derived from neuronal ensemble recordings in multiple cortical area. Two adult rhesus 
macaques (monkey K and M) were used for the study. The two monkeys were chronically implanted with arrays 
of microwires in multiple cortical areas of both hemispheres (M1, $1, and PMd). Our multi-channel wireless 
recording system was employed to sample the spiking activities from hundreds of neurons in sensorimotor cortex 
simultaneously. Both monkeys learned to navigate in a room while seated in a mobile robotic wheelchair using 
their cortical activity as the navigation control signal. Cortical ensemble recordings were converted to steering 
commands for the robotic wheelchair based on linear decoding algorithms. Both monkeys successfully acquired 
the ability to steer the robotic wheelchair towards the reward dispenser using this BMI. They were able of achiev- 
ing two-dimensional navigation with multiple starting positions and orientations of the wheelchair. 


Task Design. The monkeys were operantly conditioned to drive the wheelchair toward a food reward (grape 
dispenser). Each experimental session consisted of trials during which the monkeys navigated to the dispenser 
from one of three possible starting locations at the back of the room. Upon monkey arrival, the automated grape 
dispenser releasesd a grape (Fig. 1A). The wheelchair was then driven away from the grape dispenser to a new 
starting location and orientation. At the beginning of each experimental session we ran passive navigation trials, 
with the wheelchair executing preprogrammed routes identical from day to day. The passive trajectories were cho- 
sen empirically. Cortical neuronal responses during the passive navigation trials was used to obtain initial settings 
for two L2-norm Wiener Filters**“°, which decoded translational (forward and backward) and rotational (clock- 
wise and counterclockwise) velocity from neuronal ensemble activity. Both translational and rotational veloci- 
ties were expressed in chair-centered coordinates, not in room coordinates. Once the decoders were trained, we 
proceeded to brain-control trials during which the robotic wheelchair was steered directly by the cortical signals. 

The translational velocity was limited to —0.28 to 0.28 m/s (negative values for backward movements and 
positive for forward), and rotational velocity was limited to —46 to 46 degrees/s (negative for counterclockwise, 
positive for clockwise). If the decoded velocity exceeded the limit, the command sent to the wheelchair would be 
set to the limit values. 

The monkeys navigated in a 3.5-by-2.5 m experimental room, and the wheelchair’s dimension was 1.0-by- 
0.6 m. The actual drivable area was chosen to be 3.1-by-2.4m, to ensure safety. When the robotic wheelchair was 
at the drivable area boundary, the program would stop the wheelchair if the decoded commands would have 
moved the robot across the boundary, and would only execute the decoded commands if the robot moved inside 
the drivable area. In the task, the three starting locations had coordinates of (0.86, 0.65), (1.0, 0), and (0.86, —0.65) 
meters, where (0,0) was the room center (Fig. 1A). During brain-control trials, when the wheelchair came close 
to the grape dispenser (docking range), the program would take over the control and park it automatically close 
to the dispenser to ensure the monkeys could comfortably obtain their rewards. When the monkeys came close to 
the grape dispenser, they could reach for the food. From video tracking and wheelchair localization, we compiled 
the range of distances from the dispenser where reach was initiated. The largest such observed distance in each 
session then defined the reach zone (Fig. 1A). 


Electrode Implantation. Monkey M was implanted with four multielectrode arrays for a total of 384 micro- 
electrodes. Within each array, microelectrodes were grouped in two four-by-four, uniformly spaced grids each 
consisting of 16 microwire triplets of different lengths, for a total of 96 microelectrodes. Each hemisphere received 
two arrays, one each in the primary motor cortex (M1) and primary somatosensory cortex (S1). In the current 
experiment we used 256 channels in monkey M to record from neurons in the bilateral arm and leg areas of M1 
and S1 (Fig. S1A). 

Monkey K was implanted bilaterally with six multielectrode arrays containing 96 microwires each for a total 
of 576 microelectrodes. The arrays were implanted in the arm and leg representation areas of M1 and S1 and also 
in the bilateral premotor (PMd) cortices. In the current experiment we used 128 channels in monkey K to record 
from neurons in the $1, M1, and PMd in the right hemisphere (Fig. $1A). 


Experimental Apparatus. The experiment apparatus was composed of three components: (1) experiment 
control system (2), wireless recording system, and (3) robotic wheelchair. (Fig. S1B). The experiment control 
system controlled the experimental sequence, performed decoding of neuronal ensemble activity to wheelchair 
commands, dispensed grape, and handled the video tracking of both wheelchair and monkey head pose. The 
wireless recording system recorded neuronal ensemble activity from the monkey brain and sent the neuronal 
data to the experiment control system. The robotic wheelchair accommodated the monkey chair and received 
driving commands wirelessly from the experiment control system. The three components communicated with 
each other in a local network. 


Experiment control system. The experimental control system controlled the flow of the experiment, which 
included controlling the experimental sequence such as starting a trial, initiating docking sequence, delivering 
grape reward, and ending a trial. The experimental control system also received the neuronal ensemble spiking 
activities from the wireless recording system, computed and sent navigation (velocity) commands to the robotic 
wheelchair, performed video tracking of the wheelchair and computed the wheelchair and monkey head pose. 
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During brain-control trials, the system also ran BMI decoding algorithms that computed navigation commands 
for the robotic wheelchair from the neuronal ensemble activities. 

The location and orientation of the wheelchair were tracked using an ASUS Xtion camera (640 x 480 pixels, 
30 frames/sec) mounted on the ceiling of the experiment room. The video tracking software was written in C++ 
and built with OpenCV library. The software processed the video stream, and segmented the video frames to 
determine the robotic wheelchair location. The wheelchair orientation was determined based on the markers 
located at the front and back of the wheelchair. The wheelchair position signals were smoothed using a Kalman 
filter. Similarly, we put color markers on the cover of the monkey headcap to track its location and orientation the 
same way as we tracked the wheelchair. The position coordinate of the wheelchair was taken as the position of the 
center of the wheelchair. 


Robotic wheelchair. ‘The robotic wheelchair was a modified, commercially available mid-wheel drive model 
(Sunfire General) manufactured by Drive Medical. A Roboteq VDC2450 dual channel motor controller served 
as the interface between an onboard Raspberry Pi (RP) and the wheelchair motors. The RP received the com- 
puted motor commands through User Datagram Protocol (UDP) from the experiment control system, and the 
RP sent the commands to the Roboteq controller via serial data bus connection. As a safety feature, the Roboteq 
was programmed to stop the robotic wheelchair when the communication failed (i.e., the robotic wheelchair did 
not receive any motor commands for 1s), or hit obstacles (i.e. the wheels failed to turn as current limit was set to 
50 Amps). An emergency manual power disconnect was prominently placed on the vehicle that would disable 
the power to the wheels should a malfunction occur that requires a complete manual shut down. A secondary 
2.4GHz wireless control system also interfaced to the Roboteq controller and was used as a remote manual wire- 
less control to assist in maneuvering the vehicle between experiments when it was not receiving commands from 
the experiment control system. 


Wireless recording system. The wireless recording system was built -in-house and described in’’. Briefly, the 
wireless recording system comprised the following: headstages, wireless-to-wired bridges, and client software. 
Each headstage includes digitizing amplifiers capable of recording from 128 microelectrode channels, micro- 
processors for digital signal processing and spike-sorting after receiving templates from the client software, and 
a 2.4 GHz wireless transceiver. The bridge received incoming radio packets from the headstage and transmitted 
them via UDP to the client software on the recording computer. The client software visualized the incoming data, 
performed spike sorting, and transmitted spike-timing data to the experimental control system. 

The recording system was capable of detecting two neuronal units per recorded microelectrode channel. We 
recorded from 128 channels in monkey K and 256 channels in monkey M, yielding for each session 147.0 + 10.2 
neurons for monkey K and 157.1 + 23.9 neurons for monkey M on average. 


Data Analysis. Data description. We ran 21 sessions with monkey K sessions and 28 sessions with monkey 
M, and sometimes two sessions in a day. Two sessions of monkey K and five sessions of monkey M were excluded 
from the analysis because (1) the sessions had less than 10 attempted BMI navigation trials (3 sessions) (2), 
technical issues occurred during recording (1 session) (3), a decoder was ill-trained and biased toward negative 
velocity (1 session), or (4) communication between the experiment control system and the Raspberry Pi on the 
robot failed frequently (two sessions). Ten sessions of monkey K were further excluded from analysis because (1) 
monkey head orientation was not recorded (eight sessions), and (2) total number of valid BMI trials were less 
than 10 (two sessions). A BMI trial was invalid if the localization algorithm lost track of the wheelchair, if the trial 
was paused, and if movement errors occurred. In the end, nine sessions from monkey K and 23 sessions from 
monkey M entered the analysis. On average, each analyzed session yielded 57.9-+-/—33.0 and 45.4+/—8.6 valid 
BMI navigation trials by monkey K and M, respectively. 

Spike sorting was conducted on the first experimental session. However, neuronal units’ activity profile can 
vary between sessions due to microelectrode shifting and noise level changes. Minimal resorting was conducted 
in subsequent sessions. We excluded neurons that were not present in all analyzed sessions yielding 116 neurons 
in monkey K (27 in S1, 64 in M1, and 25 in PMd) and 124 neurons in monkey M (59 in S1 and 65 in M1). 


Position-tuning diagrams. To visualize the position-tuning properties for each neural unit (Fig. 1B-G), we first 
aligned the measured wheelchair position to the neural activities. The wheelchair position measurement was 
down-sampled to 10 Hz, and the spike counts were binned into 100 ms time-bins. Next, each unit’s spike counts 
were z-scored by the 100 ms spike-counts of that unit, then smoothed by a 3-point moving average. The room was 
divided into 5cm-by-5cm spatial bins. Then, we sorted the normalized spike counts into this spatial grid based 
on the corresponding wheelchair position, and averaged the normalized spike counts that fell into the same bin. 
Lastly, missing values of the position-tuning diagram were filled in and Gaussian-smoothed (N =7, o =3) by its 
neighboring spatial-bin values for illustration purposes. The position-tuning diagrams for different directions 
(Fig. 2A-D) were constructed similarly, where the alignment of neural activities and wheelchair position were 
conditioned on the monkey’s head orientation (a) or wheelchair orientation (3) being in a particular range. 


Mutual information. Information theoretic analysis for single neurons have been widely used to quantify the 
amount of information conveyed by the firing rate of a neuron about any experimentally measured variable or 
combination of variables'*”*. If each stimulus, s, were to evoke its own response, r, then on measuring r one would 
confirm s, and thus gain I(s) = — log,P(s) bits of information, where P(s) is the a priori probability of occurrence 
of a particular stimulus s. If instead, the same response can be invoked by several stimuli with different probabil- 
ities, this probabilistic stimulus-response relation can be expressed by a joint probability distribution P(s,r). The 
information about s gained by knowing r can be evaluated by 
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Pcs|r) 
P(s) (1) 


Averaging over different stimuli s in the set of stimuli S, the average information gain (MI) about the set of stimuli 
S present in the neuronal spike data R, (where R denotes the set of responses r) is 


I(s, R) = >, Pcrisylog, 


P(s, r) 


1(S, R) = PONS R)= Pls r)log, PPG) @) 


To obtain the amount of information a neuron’s spike train conveys about the wheelchair’s spatial location 
pre-docking, we first divided the experimental room into 0.4m-by-0.4 m grids (set of stimulus S). Wheelchair 
position measurement and spike counts were aligned and binned into 100 ms time-bins. For each neuronal unit, 
the set of responses R is the distinct spike-count observed in a 100 ms time-bin. Then, we sorted the spike-counts 
into the spatial-grid based on the wheelchair position and constructed joint-probability tables between all pairs 
of stimulus and response for each neuron. The amount of information each unit’s neuronal activity conveys about 
spatial location in bits/100 ms is then computed according to equation (2). We further bias-corrected these MI 
values following the procedures in“* by subtracting a correction-term 


1 
= ma (3) 
from each value, where Ns is the number of distinct stimulus, and Np is the number of distinct neuronal response. 
We set a neuron’s final mutual-information value to 0 if its bias-corrected mutual-information value was less than 
0, or if its C, value was greater than 1. MI for orientation parameters (a and 3) were computed similarly, with the 
set of stimulus S as a and values discretized into eight angular bins of 45 degrees each. 

We chose the position and angular stimulus discretization to ensure adequate sampling of the stimulus space - 
the number of discrete bins must not be too high for limited sampling effects, even after the correction procedure, 
to bias information estimated based on limited numbers of trials’. 

To test the significance of the mutual-information values, we performed permutation tests. We shifted the 
alignment of the wheelchair position, a or 3 a random interval between 10 to 50 seconds with respect to the neu- 
ronal activities and computed the resulting bias-corrected MI values. This was done 1000 times to obtain distri- 
butions of permuted MI values. From this permuted distribution the p-value of the actual MI value can be 
obtained for each neuron. The p-values were then corrected by false-discovery rate (FDR) and considered signif- 
icant if the corrected value is under 0.05. 


Generalized Additive Model (GAM). A generalized additive model is an extension generalized linear model 
(GLM) that involves a weighted sum of smooth functions of covariates”®. In GAM, the relationships between 
the individual predictors and the dependent variable combine generalized linear models with the benefits of 
non-parametric smoothing. Nonparametric here means that the shape of predictor functions were fully deter- 
mined by the data as opposed to parametric functions that were defined by a typically small set of parameters. 
This provides the advantage of having the ability to capture nonlinear relationships between the response variable 
and the covariates (otherwise known as “tuning function”), while still having the interpretability of a linear model. 
Mathematically, a GAM has the form of 


M 


(Ey) = Of) + etm + € (4) 


where y is the response variable, E(y) denotes the expected value, g(. ) denotes the link function that links the 
expected value to the predictors x, and x,,. The first summation contains the nonparametric terms where f are 
smooth non-linear functions describing the relationship between x,, and the response variable. The second sum- 
mation contains the linear terms and € represents the noise term — these are identical to those in a GLM. 

In GAM, each f is represented by a linear combination of spline basis functions that describes the 
predictor-response relationship as a piecewise polynomial. Penalized regression splines are used to prevent overfit- 
ting of the spline functions f . All GAMs in this work were fitted using the mgcv R-package’’. To quantify the 
dependence of firing rate of individual neurons on the different parameters, we implemented the following full 
model with mgcv: 


g(E(Y)) ~ tilx, y) + ti(v, w) + tila) + ti(9) 

+ti(x, y, v, w) + ti(x, y, a) + ti(x, y, 8) 

+ti(v, w, a) + tiv, w, 2) 

+ti(a, 2) 

S(¥(t — 1)) + S(¥(E + 1)) (5) 


The response variable Y is the number of spikes in 100 ms time bins, Y(t — 1) and Y(t+ 1) represent the num- 
ber of spikes shifted in time by one time bin. The link function g(-) was chosen to be log and the response was 
fitted assuming quasipoisson conditional distribution. The predictors x, y, v, w, a, and 3 represent the wheel- 
chair’s x-position, y-position, linear velocity, rotational velocity, monkey head angle and wheelchair angle, respec- 
tively. The functions S() and ti() indicate nonparametric smoothing is to be used. ti() was used because both the 
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main-effects and the interaction between the main-effects were included in the model. In equation(5), the terms 
on the right-hand stand of the first line represents the main-effects of position, wheelchair velocity, head, and 
wheelchair angles, respectively - these can be seen as the “tuning function” or “tuning curve” of the neuron to 
these predictors. Note that the effects of both position and velocity main-effects are to be modeled with 2D 
smoothing functions, as indicated by the bivariate functional form. The second line of equation (5) represents all 
the interaction terms between position and the other three predictors. Since the main-effect of position is 
assumed to be a bivariate function, these terms represent the interaction between a bivariate function with either 
a bivariate (e.g. ti(x, y, v, w)) or univariate (e.g. ti(x, y, a)) function. The third and fourth line of equation (5) 
include all the interaction terms between velocity and orientation parameters, and between the orientation 
parameters, respectively. The last line of equation (5) includes the effects of previous and next bins of spike-counts 
- these terms are included for a more complete model specification because spike-counts are known to and 
indeed exhibited autocorrelation”. This full model specification was deemed appropriate after examining diag- 
nostic plots. 

To assess the relative importance of a group of covariates, we fitted partial models from the full model by 
removing those covariates. For example, to assess the effects of position on a neuron’s activities, we constructed 
the partial model by removing the ti(x, y) term from equation (5). F-test was conducted to check if the full-model 
is significantly better than the partial-model, i.e. if a neuron is “tuned” to position. To quantify the effects of posi- 
tion, or tuning-depth (TD), we calculated the McFadden’s pseudo R-squared”! as 

1 — 22 
Dp (6) 


where D, and D, are the deviance of the full and partial-models, respectively. Deviance is analogous to the sum 
of squares of residuals in regression by ordinary least squares (OLS) and measures the lack-of-fit of a model. The 
pseudo R-squared quantity is analogous to partial R-squared in OLS and measures how much of the neuronal 
activities can be explained by position. The full and partial model differ only in the main-effect term of position, 
and the interaction terms between position and all other predictors are kept so that equation (6) measures only 
the contributions to the neuronal activities from position alone, excluding those due to predictor correlations. TD 
to position, head-direction a and wheelchair direction (3 were calculated this way for all neurons. 


Place-field heatmap. The GAM fitting process produces the neurons’ position tuning-functions as the bivariate 
position main-effect terms. For each neuron, we defined its place field as the room locations where the neuron’s 
tuning function exceeded the median value over all room locations. A histogram of place-fields was constructed 
from all position-preferring cells’ place-fields and normalized by the number of position-preferring cells in each 
session. These session histograms were then averaged to produce the final heatmap. 


Preferred-direction histogram. The GAM fitting process produces the neurons’ a- and 3- tuning curves as the 
univariate cyclicala and 3 main-effect terms. For each neuron, we defined its preferred directions as the argmax 
of the tuning curves. Histograms of the preferred directions were constructed from all orientation-preferring 
cells’ tuning curves and normalized by the number of orientation-preferring cells in each session. These session 
histograms were then averaged to produce the final histogram. 


Consistency of tuning properties. To analyze the consistency of individual neurons’ tuning properties within a 
session and across multiple sessions, we first excluded units that were resorted in any of the analyzed sessions. 
Twelve cells were excluded in monkey K and 15 cells were excluded in monkey M. We then selected from each 
monkey neurons that were highly tuned to either spatial location or orientation in at least 70% of analyzed ses- 
sions. For each neuron, we constructed a three-dimensional histogram where each bin represents a unique com- 
bination of (x-position, y-position, head-direction «). The neuronal activities falling within each histogram bin 
were then averaged. This trivariate tuning function allowed us to compare position- and orientation-tuning prop- 
erties at the same time. To ensure adequate sampling, we used 0.1 m discretization for x- and y-positions, and 
divided head-direction into four bins of 90 degrees each. 

To evaluate how similar the tuning properties were between two sessions, we obtained the pair-wise con- 
sistency index by first calculating the Pearson correlation between each neuron’s trivariate tuning function in 
the corresponding sessions and then averaged that from all neurons. The within-session consistency index was 
calculated similarly by dividing each session into two equal continuous blocks and obtaining a trivariate tuning 
function for each neuron in each block. 

To calculate the significance of the consistency index, we performed permutation tests by shuffling the ses- 
sions’ neuronal data by a random time between 10 and 50 seconds with respect to the covariates and computing 
all pairwise- and within-session consistency indices. This procedure was repeated 1000 times to obtain distri- 
butions of permuted consistency indices. From this permuted distribution the p-value of the actual consistency 
index value was obtained. To compare pairwise- and within-session consistency of tuning properties, we per- 
formed the Wilcoxon rank-sum test between pairwise- and within-session consistency indices. The consistency 
indices were considered to be statistically different if the test rejected the null hypothesis at alpha level of 0.05. 


Off-line position and orientation decoders. For monkey K, nonlinear autoregressive neural network with external 
input (NARX) decoders were trained and tested off-line to predict position, head-direction a and wheelchair 
direction 3. NARX differ from the common feed-forward neural network in that the decoded outputs are fedback 
to the input layer°’. This architecture is suitable for time-series prediction because it utilizes the information 


SCIENTIFIC REPORTS | (2018) 8:9184 | DOI:10.1038/s41598-018-27472-4 14 


www.nature.com/scientificreports/ 


present in the data’s autocorrelation structure. Position decoders had a single hidden layer with 30 artificial neu- 
rons (nodes) fully connected to two output nodes corresponding to the x- and y-position. a and ( decoders each 
had a single hidden layer with 10 nodes fully connected to two output nodes corresponding to the sine and cosine 
of the target angle. Decoder output at time t was computed from five 100ms-bins of z-scored spike-counts (each 
bin correspond to time t tot — 4) of the chosen neurons, and three past predictions (time t — 1 tot — 3). For 
monkey M, feed-forward neural network (FFN) decoders were used where the decoded outputs were not fedback 
to the input, and all other network configurations stayed the same. We found this produced less noisy predictions 
for monkey M. The NARX and FEN decoders were implemented using MATLAB 2014b’s (Mathworks, Boston, 
Massachusetts) neural network toolbox. All hidden layer nodes had hyperbolic-tangent transfer function, the 
output layer nodes had linear transfer function. All decoders were trained with conjugate gradient descent with 
Fletcher-Reeves updates” with early-stopping after 15 validation failures. 

For each session we performed 10-fold cross-validation on the brain-controlled trial data pre-docking with 
five retrainings per fold, yielding for each fold a best performing decoder for each parameter. The performance 
was measured by either Pearson's correlation coefficient (CC) between the predicted and actual values for orien- 
tation parameters, or mean prediction error (MPE) for position. MPE is defined as the average distance between 
the predicted and actual locations. The median CC (Fig. S4) or MPE (Fig. 5A) across all folds were taken as a 
session's decoding performance. 

To assess the chance decoding performance, the decoders were trained and tested on shuffled session data 
constructed by first shifting the neuronal data by a random amount with respect to the covariates, and then 
shuffling the resulting time series (to destroy the autocorrelation structure). For each session-population-type 
combination, this was done 100 times to obtain 95% confidence intervals for chance decoding performance, from 
which we obtained the p-value of actual decoding performance. 

To visualize the location prediction performance over space (Fig. 5B-G), we first concatenated the prediction 
error, defined as the distance between the predicted and actual locations, from all 10 folds of testing data. These 
prediction errors were then sorted into the spatial-grid defined in position-tuning diagram and averaged within 
each bin. Lastly, spatial bins with missing values were filled in and Gaussian-smoothed (N = 5, g =2) by the val- 
ues of its spatial-bin neighbors. 


Reach activity. Population peri-event time histograms (PETHs) aligned on the wheelchair entering the reach 
zone was constructed from smoothed z-scored spike-counts (100 ms time bins, 3-point moving average) from all 
brain-controlled trials in a session. Neuronal units were sorted by the slope of its peri-event activities, as calcu- 
lated by ordinary linear regressions (Fig. SsA-B). 


Reach decoder. For brain-controlled trials in each session, we labeled all neuronal activities inside the reach-zone 
as reach-related, and all neuronal activities outside as navigation-related. There was roughly nine times as many 
navigation-related time points as reach-related in each session, making the data-set highly imbalanced. Principle 
component analysis (PCA) was first conducted on the smoothed z-scored spike-counts (100 ms time-bins, 
3-point moving average). Ensemble decision tree classifiers were then used to classify reach activities from the top 
five PCs. Each ensemble classifier had 400 decision trees®* with minimum leaf-size of 20, and was trained using 
MATLAB 2014bs fitensemble function. Random under sampling (RUSBoost)*™ was chosen to fit the ensemble 
classifiers. For each session, we performed 5-fold cross-validation, yielding for each fold an ensemble classifier. 
The performance of each classifier is measured by Cohen's k”*? between the actual and predicted labels (reach or 
not) on the testing data. Cohen’s « takes into account of the distribution of the actual labels and is an appropriate 
measure for imbalanced data classification. The « values for all five folds were averaged to obtain each session’s 
reach-classification performance. 

To visualize reach-classification in space (Fig. SsC-H), we first concatenated the predicted probabilities 
(scores) of reach from all five folds of testing data in a session. The score of reach at each time point is the mean 
terminal leaf probability across all 400 decision-trees in the ensemble classifier at that time. We then sorted the 
scores into the spatial-grid as defined in position-tuning diagram and averaged the scores that fell into the same 
bin. Lastly, spatial bins with missing values were filled in and Gaussian-smoothed (N =5, 0 =2) by the mean 
scores of its spatial-bin neighbors. 


Statistical analysis. To assess the difference between preferred TDs among brain regions within the same 
monkey, we used the Kruskal-Wallis test with Tukey’s multiple comparison on neurons pooled over all sessions. 
To assess the difference in proportion of neurons tuned to space or orientation among brain regions within the 
same monkey, we used Friedman’s test with Tukey’s multiple comparison, where sessions were a repeated meas- 
ure. To assess the difference between preferred TDs between different monkeys, we used Wilcoxon rank-sum test 
on neurons pooled over all sessions for each monkey. To compare the position and orientation decoding perfor- 
mance from using different neuronal populations, we used Kruskal-Wallis test with Tukey’s multiple comparison. 
To assess the uniformity of the population place-field-heatmap (Fig. 3A,B) and preferred-direction histograms 
(Fig. 3C-F), we used the Kolmogorov-Smirnov test to compare those distributions against the uniform distribu- 
tion. To assess the decoding of reach-state taking into account o the ensemble size for reach state decoding, we 
used nonparametric ANCOVA. We consider a statistical test to be significant if p < 0.05, and all Tukey’s multiple 
comparison had a level of 0.05. 
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